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Executive Summary 



The National Literacy Secretariat (NLS) is a grants and contributions initiative established in 
1987. The NLS has two objectives: 

• To increase opportunities and take-up, so that people improve their reading and writing 
skills; and 

• To work towards making Canada’s social, economic and political life more accessible 
to those with weak literacy skills. 

The NLS pursues a partnership approach with the provinces, territories, non-profit 
groups, and business and labour organizations. The NLS works with and through its 
partners to enhance literacy through two funding streams: a national funding stream to 
provide funding support to help address literacy issues that have a national scope; and a 
federal/provincial/territorial (FPT) funding stream to provide funding for community 
based projects that address regional and local needs. 

From 1997/98 to 2001/02, the NLS distributed an average of $29 million per year in 
grants and contributions to help create a more literate Canada. The NLS allocates the full 
amount in grants but converts some of the funds into contributions on a case-by-case 
basis in accordance with Treasury Board policy. 

The NLS was last evaluated in 1995. The 1995 evaluation concluded the NLS was working 
in areas not dealt with by the provinces, had made a significant contribution to literacy in 
Canada (e.g. by encouraging many sectors become involved in literacy, reducing barriers to 
hteracy training for some groups), and was highly rated by program sponsors. 

Regarding areas for improvement, the 1995 evaluation noted that the NLS had not been 
proactive in terms of specifying priority areas for funding, and that literacy issues had not 
been well defined for the purpose of federal intervention. The 1995 evaluation also noted 
that the NLS should play a more active role in ensuring the quality of research and 
learning materials produced with its funding, improve the dissemination of information, 
and establish program safeguards to ensure against project duplication. 

Since the 1995 evaluation, there have been a number of changes in the environment of the 
NLS. For example, the shift towards an interdependent, globalized and knowledge-based 
society is creating new challenges for literacy. In addition, an increased emphasis on 
public accountability means that organizations that distribute public funds are being 
asked to continually review and assess their performance to ensure their relevance, 
probity and effectiveness. 

The current evaluation of the NLS covers the five-year period from 1997/98 to 2001/02. 
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Approach to the Evaluation 

In accordance with the objectives of the NLS, the purpose of the evaluation is to study 
the effects of the NLS both, in terms of the opportunities for improved literacy and in 
terms of working towards making Canada’s social, economic and political life more 
accessible to people (i.e. benefits to individuals) with weak literacy skills. While the 
creation of opportunities is directly linked to the activities of the NLS, actual benefits to 
individuals may be viewed as an indirect, albeit and essential consequence of the first. 

Evaluating partnership programs, such as the NLS, presents a number of practical 
challenges. As a partnership program, the NLS is designed to achieve its objectives 
through organizations and groups that are beyond the direct influence of the Secretariat. 
Also, the NLS funds a diverse range of projects and outputs, and there are no common 
metrics (such as the number of job placements) by which performance can be quantified. 
Therefore, in specific instances, it is difficult to track and measure program results/effects. 

In addition, most (88% in 2001/02) of NLS funding is provided through grants and, 
by definition, grants have limited accountability requirements. 

Given the nature of the relationships between the NLS and its partner organizations/projects 
(grant based) and the nature and scope of the projects, detailed level data are not available for 
direct estimation of effects on individuals. For example, some of the projects are concerned 
with the development of literacy related products while others that provide services directly 
to individuals, may focus on subsets of the population - relatively small groups (compared to 
the general population - making it difficult if not impossible to detect effects on the basis of 
general surveys / census based on the general population). Consequently, the focus of the 
evaluation is on the extent the organizations succeeded in providing expected products and, 
indirectly, qualitative assessments regarding the likely impacts on the individuals assisted. 

The methodology used to evaluate the NLS recognized these challenges and attempted to 
address them in a number of ways. For example, the evaluation took a comprehensive 
approach and included a wide range of stakeholders related to the NFS and the literacy field. 
The use of multiple lines of evidence was emphasized to allow for findings from 
one approach to substantiate/corroborate and inform other lines of evidence. Case studies 
were used to provide in-depth analysis/illustrations of program issues, outputs and impacts. 

The evaluation approach includes the following components: 

• Program database review: A review of the NFS database was used to develop an 
understanding of the scope of NFS funding activities and as a starting point for 
developing questions to be explored by other lines of evidence; 

• Case studies: Twenty-one case studies were completed (three funded under the 
national funding stream, and 18 funded under the FPT funding stream); 

• Key informant interviews: A total of 43 key informant interviews were conducted with 
5 stakeholder groups: NFS representatives (3), federal government representatives (4), 
representatives from organizations that received NFS funding from one of the NFS 
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funding streams (10), provincial and territorial representatives (13), organizations that had 
applied but were not accepted for funding from one of the NLS funding streams (13); 

• Survey of NLS funded projects: A telephone survey was undertaken of 
277 representatives of NLS funded projects. The survey was designed to delve deeper 
into issues and further confirm findings that emerged from the case study reviews and 
the key informant interviews; 

• Focus groups: To fill in gaps and further explore issues emerging from the other lines 
of evidence, two focus groups were conducted (one composed of experts in the literacy 
field; and one with representatives from organizations that face the same challenges as 
the NLS in areas of funding provision, partnership activity and performance 
measurement). Focus group participants did not have a direct relationship with 
the NLS. 

Evaluation Findings 

Impacts and effects : There is evidence that NLS funding has improved literacy 
opportunities for a range of groups. 

The case study analysis and key informants indicated that the NLS has had an important 
impact in areas such as family and workplace literacy. The project survey indicated that 
NLS funded projects improved literacy opportunities for a range of groups including low 
income people, people with disabilities, first generation Canadians, and Aboriginal 
people. 

A catalyst : The general view is that the NLS acts as a catalyst to increase awareness 
of literacy issues, bring people together and advance literacy. 

Evidence from case studies, key informant interviews, and the focus group of literacy 
experts indicates that the NLS has raised the profile of literacy and helped to increase 
collaboration/partnerships within the literacy community. The available evidence also 
suggests that the NLS has advanced the literacy movement by promoting research and 
improving the ability to deliver successful literacy programs within the community. 

Also, there is a high level of satisfaction with 70% of the organizations surveyed 
indicating that they were either satisfied or very satisfied with NLS services. 

Partnership model : Regarding the FPT funding stream, the partnership model has 
been functioning well in the Canadian inter-jurisdictional environment. The NLS 
should consider, however, improving communication with partners in relation to the 
selection and announcement of projects funded under the national funding stream. 

All provincial and territorial stakeholders, as well as funded organizations familiar with 
the NLS and its activities, were satisfied with the functioning of the partnership model as 
it applied to the LPT funding stream. Lor example, they cited that it facilitated flexibility 
and openness. While the NLS partnership model is viewed as working well under the 
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FPT program, provincial and territorial partners would like to see better communications 
and working relations with the NLS in the case of the program’s national funding stream. 

Project funding ; NLS funding is broadly dispersed across many organizations. 

Over 1997-2002 period, 2043 organizations received NLS funding for 2,853 projects. 
Most organizations (80%) had only one project during the period. In terms of funding 
support, 50% of all projects funded by the NLS received less than $25,000. 



Areas for Improvement 

Areas for improvement identified in the 1995 evaluation included the need for clearly 
defined priorities concerning issues to be pursued and projects funded, a more active role 
in ensuring the quality of research and learning materials produced and improved 
dissemination of results in order to safeguard against project duplication. Although 
program enhancements have occurred since 1995, (e.g. improved communication) there 
is still room for improvement (e.g. monitoring of results). 

Leveraging ; There is some evidence that NLS support results in organizations being 
able to leverage further resources to support literacy efforts. However, there is a 
need for the NLS to be able to demonstrate that it has program safeguards in place 
to ensure that NLS projects and leveraged funds are incremental. 

The majority of organizations that receive funding from the NLS also receive funding 
and/or in-kind resources from project partners and over time the share of total project 
funds accounted for by partners’ contributions is declining. The program needs 
safeguards in place to protect against projects being funded that would have proceeded 
regardless of NLS funding. A potential safeguard could require sponsors to attest on the 
project application that the project would not proceed without NLS support. 

Dissemination of information ; While the NLS is credited for supporting the National 
Adult Literacy Database (NALD) and International Adult Learning & Skills Survey 
(lALSS) as well as regular intergovernmental consultation, more attention should be 
given to ensuring dissemination of project results among stakeholders. 

The majority of the provincial and territorial key informants indicated that 
communication across the country has increased significantly over the last five years 
through the activities of the NLS, leading to the dissemination of innovative ideas 
throughout the literacy community; however; evidence from the case studies and key 
informants indicate that there is a need to further improve the dissemination of 
information and results to stakeholders and partners. 

The financial support for the NALD and the lALSS provided by the NLS reflects the 
commitment of the NLS to improve communication and sharing of data. At the same 
time, however, the evidence indicates that there is a need to further improve the 
dissemination of information and results to stakeholders and partners. 
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Overlap and duplication ; The general view is that as a catalyst and an 
organization that works in partnership to achieve its objectives, the NLS plays a 
unique role that is not being duplicated. There is a risk, however, of duplication in 
specific project support. 

There is a risk of project overlap and duplication, particularly in the application of the FPT 
funding stream. Evidence from case studies, key informants and survey respondents indicate 
a risk of duplication in specific project support. This risk arises from the current lack of 
formal or strategic dissemination of information regarding the results of previous projects, 
and a current lack of information sharing on project proposals that are under consideration. 
Increased sharing of information with NLS partners was identified as a way to help ensure 
that a research project in one part of the country is not duplicated in another. 

Program implementation ; The NLS could simplify and streamline procedures to 
make it easier for smaller organizations less familiar with the grant process to 
obtain funds. 

Groups familiar with the NLS and its application process feel that process and guidelines 
are clear and work well. Organizations that are less fa mi liar with the NLS and its work 
find that application and notification procedures are cumbersome, that forms are complex 
and that the need to develop new annual applications and approaches makes it difficult 
for smaller organizations to be successful in the competition for project funding. 

Quality : The NLS should take steps to ensure the reliability and credibility of 
research projects. 

While major research projects conducted through SSHRC and Statistics Canada ensure 
high quality and reduce the risk of being duplicative through peer review mechanisms, 
LPT research runs the risk of duplicating work done in other regions/communities. 
In addition, more than one-half of the provincial and territorial key informants felt that 
there is insufficient program monitoring and accountability on the LTP funding stream 
to ensure the quality of products. Lurthermore, needs assessment projects funded under 
the LPT stream may lack the rigour necessary to be considered ‘applied research’. 
The application of a formal peer review process for all research conducted would help 
increase quality and help reduce the risk of duplication. 

Monitoring performance : The NLS should put in place efficient data gathering and 
monitoring systems and consistent procedures for evaluating projects. 

The monitoring of the results of projects by the NLS was identified as an area for 
improvement, even given the limitations imposed by the grant process. Also, there is a 
need to improve the data gathering and monitoring system (GMAX) used for the 
administration of grants. Coding limitations restrict its usefulness in providing an in- 
depth perspective of NLS activities and outputs. Data collection and storage should be 
examined to ensure that data is available to conduct a future evaluation. 

While the NLS is credited with being a catalyst in promoting literacy across Canada, it is 
unclear to what extent it has been successful in meeting its stated objectives, due in part 
to issues of attribution and measurement. 
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V 




Relevance ; While the NLS plays a unique role in promoting literacy across Canada, 
there may be an opportunity to develop a more strategic approach through the 
setting of focused priorities. 

There is a concern as to what the organization’s role in the future should be relative to its 
current objectives and the needs of the literacy community. One approach is for the NLS 
to ‘stay the course’. This approach reflects the view that more work needs to be done in 
raising the profile of literacy, and that the NLS as a funding organization continues to 
play a unique role in supporting the attainment of its objectives through other 
organizations. Another approach is for the NLS to consider developing a national 
approach to literacy that would involve the setting of Canada-wide priorities that would 
reflect the specific needs of today (e.g. focusing on technology, focusing on the needs of 
specific groups such as Aboriginals and families). 

This suggests that there is an opportunity for the NLS to re-examine its role in order to 
determine how it could best meet its objectives in the changing environment. This could 
include a consultation process with stakeholders across the country to assess how the 
literacy challenge has evolved over the last fourteen years. 

Sustainability and NLS Role : As part of developing a more strategic approach, 
the NLS may wish to look at ways to increase the sustainability of results and progress. 

The NLS is the primary literacy organization in Canada which funds and supports 
first-time projects. More than half of the NLS funded projects continued after program 
funding expired, however, almost one-third of NLS funded projects were not sustainable. 
There are some pressures, however, for the NLS to consider providing longer-term 
funding in order to promote, expand and build upon past successes and to improve on 
project sustainability. While this could raise inter-jurisdictional issues, such an approach 
could be considered as part of a re-examination of the program’s role. 
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Management Response 



The National Literacy Secretariat (NLS) would like to thank participants who gave their 
time for this evaluation which was conducted over the summer and fall of 2002. In this 
Management Response, the NLS outlines areas of strength and acknowledges where 
improvements are warranted. 



BACKGROUND 

The National Literacy Secretariat (NLS) was established in 1988 as a result of increasing 
awareness and evidence that Canada was facing a serious challenge in the area of low literacy 
skills. Since, the NLS has worked to promote literacy as a foundational component of 
learning and to make Canada’s social, economic and community life more accessible to 
people with weak literacy skills. 

The key activities of the NLS support work which develops adult literacy learning 
materials and models, improves outreach and access to literacy programs, increases 
public awareness of the importance of literacy, supports research, and improves 
coordination and information sharing among literacy stakeholders. In carrying out its 
mandate, the NLS partners with the provinces and territories, non-profit organizations 
(NGOs), post- secondary institutions, provincial/territorial institutions (including crown 
corporations), as well as business and labour organizations. The NLS also works with 
many federal departments and agencies, particularly in the areas of Justice, Corrections 
and Health, to encourage horizontal support for literacy issues. 



ISSUES IDENTIFIED BY THE EVALUATION 

Monitoring, Accountabiiity and Resuits Management 

The Treasury Board Policy on Transfer Payments has one requirement for the use of grants 
as a funding mechanism which is that a program must verify the on-going eligibility of any 
funding recipient. This is quite different from contributions, which can be audited, and must 
provide financial statements, outcome data, interim and final reports. 

The NLS receives its funds in the form of grants and has historically used its authority to 
fund most projects with this mechanism. Contributions have been used when there are 
certain risk factors involved such as a large sum of money or a new recipient without a 
track record known to the NLS. Joint initiatives with other government departments would 
also require the use of contribution agreements. In spite of Treasury Board requirements 
for using grants, however, the NLS has managed grants in a similar way to contribution 
agreements by requiring financial statements, and interim and final reports. In addition, 
unused grant funds are returned to the NLS, to efficiently use scarce resources. 
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Nonetheless, the current evaluation found a need for improved accountability in the NLS 
management of grants and contributions. In response to this the NLS will strengthen its 
requirements through an increased use of contribution agreements, while recognizing that 
there will always be a need for the use of grants particularly for those projects which are 
low-risk and support very small community organizations. The NLS has discussed 
the issue with Treasury Board and will use a phased-in approach, initially implementing 
the change with larger organizations and projects. 

A new NLS evaluation framework and performance measurement strategy are currently 
being developed. The performance measurement strategy in particular will indicate specific 
data which the NLS will require from its clients in the future. The NLS has undergone a 
process of developing a new logic model, on which the evaluation framework is based, 
as well as a newly articulated program objectives. The evaluation framework will include a 
formative evaluation in year three and a summative evaluation in year five. The strategy 
will include some in-depth case studies designed to target the end-user in order to 
more effectively measure and capture the impact of the programming on individuals. 
This framework is expected to be completed by the end of September 2004. 

The evaluation report states a need for an improved system for data gathering and 
monitoring. As with other programs, the NLS now uses the departmental Common 
System for Grants and Contributions (CSGC) as its main system for data gathering. 
In addition to this, the NLS has initiated discussions with key stakeholders regarding the 
kinds of data they collect and what might be possible to share. 

Relevance 

The 2002 evaluation report reaffirms that the Secretariat continues to play a unique role 
within the literacy community and that the five eligible funding areas that support 
capacity development remain relevant. Evaluation participants referred to the need for 
nation-wide leadership, a national literacy campaign, and the expansion of the role of the 
Secretariat in bringing together provinces and territories. 

Several other initiatives confirm the relevance of the work of the NLS: 

• The ongoing concern with literacy levels, supported in part by the International Adult 
Literacy Skills Survey (lALSS) currently underway, also demonstrates the relevance of 
the program. Country results are expected in February 2005 and in-depth Canada 
results will follow in Spring 2005; 

• The National Summit on Learning and Innovation which took place in November 2002 
resulted in a series of recommendations. One of these was for the development of a 
Pan-Canadian literacy strategy; 

• The Standing Committee on Human Resources Development and the Status of Persons 
with Disabilities undertook to study adult literacy in Canada in 2003. Its work resulted 
in a report entitled Raising Adult Literacy Skills: the Need for a Pan-Canadian 
Response which contained 21 recommendations. Human Resources and Skills 
Development Canada (HRSDC) led the Government of Canada response to the report 
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and in doing so, committed to engaging partners in discussions to work towards a Pan- 
Canadian strategy on literacy. 

Partnership 

The Secretariat is highly regarded for its partnership approach as indicated in both the 
1995 and 2002 evaluations. All provincial and territorial stakeholders, as well as funded 
organizations familiar with the Secretariat and its activities, believe that the partnerships 
function well within the federal/provincial/territorial (FPT) funding stream. In addition, 
the Secretariat structures individual working partnerships designed to meet the needs of 
each province and territory. 

The NLS evaluation found that communication needs to be improved within the national 
funding stream. This is being accomplished through improved and increased web-based 
communication. Since the 2002 evaluation, calls for proposals have been issued for both 
the Research Partnerships and the National Non-Literacy projects funded under the 
national funding stream. In addition, a research consultation was held in March 2004 
with stakeholders and partners from across the country. A range of issues was discussed, 
including accountability in literacy research and criteria for quality standards. A research 
framework is being developed based on the consultation and will be completed in 
September 2004. This framework will guide research funding priorities and principles 
for the NLS. 

Leveraging/Incrementality 

The NLS requires that funding applicants provide evidence of additional funds already 
secured through other partnership(s) when a proposal is submitted. NLS records indicate that 
this has resulted in over $200 million from external sources being contributed to its funded 
initiatives since 1988. The evaluation report indicated that the majority of the larger 
provinces now contribute more funds to literacy than required by the NLS partnership model. 
For example, British Columbia recently doubled its funding for literacy. 

The evaluation report confirmed that NLS funding results in organizations being able to 
leverage further resources to support on-going literacy efforts. Evidence of incrementality 
(the extent to which the program activity or interventions would have taken place regardless 
of any NLS support), however, was cited as a weakness in the evaluation report. 

The NLS agrees that there is a need to demonstrate that it has program safeguards which 
would help to ensure that no other source of funds are being accessed for the same project 
or being displaced. Therefore during the 2004-2005 fiscal year the NLS will put in place 
measures to have funding applicants attest that NLS support is required for their project 
to be undertaken, and that the project will not be able to take place without the funding 
support of the NLS. Specifically, protocols will be included on the application form itself 
for clients to declare that their project could not be undertaken without the assistance of 
the NLS. 
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IX 




Information Dissemination 



The majority of provincial and territorial key informants in the evaluation considered 
communication across the country to have increased significantly over the last five years 
through the activities of the NLS. They believe that this has led to the dissemination of 
innovative ideas throughout the literacy community. 

Since the previous evaluation, additional information dissemination measures were 
undertaken. The NLS now: 

• requires that all NLS-funded project descriptions be posted on the National Adult 
Literacy Database (NALD) web-site; 

• provides all products to the HRSDC library which is accessible to the public and whose 
catalogue is available on-line via the NALD; the NLS requires at least 2 copies of all 
projects and their products to be filed at provincial and territorial literacy resource 
centers to ensure that NLS-funded material has adequate distribution; 

• is exploring the feasibility of putting all products/materials on NALD as full text 
documents (currently materials are put on NALD in full text on an ad hoc basis); 

• supports the literacy database at the University of Alberta; 

• funds a number of newsletters such as Connect and Literacy.ca and supports a new 
literacy research journal to publicize materials and resources developed; and 

• supports conferences as a vehicle for sharing information, such as the National Best 
Practices Workshop on Literacy. 

Overlap and Duplication 

Federal/provincial/territorial literacy funding processes have typically included project 
review committees. One of their functions is to ensure that projects recommended for 
funding are not duplicated. The proposal review committee in each province or territory 
typically includes representation from the NLS, the respective province or territory, as well as 
individuals from the NGO or academic community that have additional expertise. 

The evaluation did not find evidence of overlap or duplication, but in the absence of 
specific measures to prevent this, it identified the area as being in need of improvement. 
Further measures to avoid overlap and duplication will be taken by the NLS, requesting 
that applicants attest to having done literature searches related to their issue/proposed 
activity on NALD and an environmental scan to ensure that no current initiatives exist 
that would duplicate their efforts. 



X 
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CONCLUSION 



Strengths 

The evaluation found that the NLS had an important impact in areas such as family and 
workplace literacy, and that it improved literacy opportunities for a range of 
disadvantaged groups such as Aboriginal people and people with disabilities. As a 
catalyst, the NLS was found to increase awareness of literacy issues and to increase 
collaboration and partnerships in the literacy community. It has advanced research on 
literacy and improved the ability of organizations to deliver successful literacy programs 
within the community. Evaluation participants also expressed a high level of client 
satisfaction with the NLS and its program. The federal/provincial/territorial funding 
stream, in particular, was pleased with the partnership approach used by the NLS. 

Recommended Areas for Attention 

The evaluation report identified a number of areas that require attention. Increasing the 
proportion of contributions will strengthen the NLS’s ability to monitor projects and 
improve its ability to demonstrate outcomes. The evaluation framework and performance 
measurement strategy will also assist in fine-tuning the program’s ability to track 
outcomes. Safeguards are required to prevent the funding of projects that could go ahead 
without NLS funds. Mechanisms are needed to avoid duplication and overlap. Application 
procedures need to be simplified and streamlined. We must ensure the quality of research in 
the federal/provincial/territorial funding stream in order to improve the program. Data 
gathering and monitoring systems also need improvement. 

Work has begun in several of these areas 

The work of the NLS goes beyond the administration of grants and contributions, which 
much of the evaluation covers. It is intended that future evaluations will provide a clearer 
view of the value added of the NLS in helping to build the literacy infrastructure that Canada 
needs, as well as its strategies of partnership development and multi-sectoral targeting. 
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ADDENDUM TO THE 2002 NATIONAL LITERACY PROGRAM 
(NLP) EVALUATION 

Since the 2002 evaluation, changes have been implemented in response to evaluation 
findings. For example, a regular call for proposal process has been developed and 
implemented. A research consultation on adult literacy was held with experts from 
across the country to inform a new literacy research framework. A new evaluation 
strategy was developed. Funding applicants must now attest that NLP support is 
necessary to the undertaking of their project. Applicants must also conduct a complete 
literature search of relevant web sites in order to prevent the duplication of any 
undertaking. The findings of both NLP evaluations, in addition to the results of the above 
consultations, have provided support to program design and implementation. 

Since 2002, several key events have taken place which have influenced the context in 
which the NLP conducts its activities. These include: 

• The 2003 hearings of the Standing Committee on Human Resources Development and 
the Status of Persons with Disabilities on adult literacy and the subsequent release 
of its Report entitled, '"Raising Adult Literacy Skills: the Need for a Pan-Canadian 
Response 

• The 2005 commitment of an additional $30 million to the NLS for the further 
development of partnerships with provinces and territories, business and labour to 
foster awareness of and involvement in literacy issues and to promote learning in the 
workplace; 

• The 2005 cross-country community consultations on learning, literacy and essential 
skills; and business and labour consultations on literacy and essential skills; 

• The 2005-06 conduct of an interim evaluation of the NLS required by Treasury Board 
Secretariat prior to approval of any further terms and conditions; 

• The 2005 release of the results of the 2003 Adult Literacy and Life Skills Survey, and 
cross-country presentations promoting the results as well as dialogue and collaboration 
on future actions; 

Enhanced performance measurement and results reporting guided by a strong evaluation 
framework will continue to be a priority in implementation of learning and literacy 
programming. 
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1. Introduction 



The National Literacy Secretariat (NLS) is a grants and contributions initiative established in 
1987. The NLS has two objectives: 

• To increase literacy opportunities and take-up, so that people improve their reading and 
writing skills; and 

• To work towards making Canada’s social, economic and political life more accessible 
to those with weak literacy skills. 

The NLS pursues a partnership approach with the provinces, territories, voluntary groups 
and associations, and business and labour organizations. Rather than being involved in 
direct program delivery, the NLS works with and through its partners to enhance literacy. 
From 1997/98 to 2001/02, the NLS distributed an average of $29 million per year in 
grants and contributions to help improve literacy in Canada. 

The NLS was last evaluated in 1995. The 1995 evaluation concluded the NLS was 
working in areas not dealt with by the provinces, had made a significant contribution to 
literacy in Canada (e.g. by encouraging many sectors and organizations to become 
involved in literacy, reducing barriers to literacy training for some groups), and was 
highly rated by program sponsors. 

Regarding areas for improvement, the 1995 evaluation noted that the NLS had not been 
proactive in terms of specifying priority areas for funding, and that literacy issues had not 
been well identified for the purpose of federal intervention. The 1995 evaluation also 
noted that the NLS should play a more active role in ensuring the quality of research and 
learning materials produced with its funding, should improve the dissemination of 
information, and should establish program safeguards to respond to issues of project 
overlap and duplication.' 

Since the 1995 evaluation, the environment impacting on the NLS and the manner in 
which the federal government manages programs has changed. The shift towards an 
interdependent, globalized and knowledge-based society is creating new challenges for 
public, private and non-profit players involved in literacy. In addition, the federal 
government has further increased its emphasis on public accountability by requesting that 
organizations that distribute public funds continually review and assess their performance 
to ensure their relevance, probity and effectiveness. 

The current evaluation of the NLS covers the five-year period from 1997/98 to 2001/02, 
and addresses issues of program relevance, design and delivery, achievement of objectives 
and impacts. 



’ The management response for the 1995 evaluation indicated that an effort would he made to more clearly articulate 
priorities without harming program flexibility, monitoring procedures would be codified, improvements had been 
made to the tracking and dissemination system, and consideration would be given to ensuring that an organization 
which completes a project has funds attached to it to ensure distribution. 
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This report on the current evaluation includes the following sections: 

• An overview of the NLS; 

• A summary of the evaluation issues and methodology; 

• An overview of projects funded by the NLS, and an examination of the impacts and 
effects of NLS funding; 

• A consideration of the NLS’s success at achieving its objectives; 

• An assessment of implementation issues; 

• A review of the rationale and relevance of the NLS; and 

• A summary of the main findings and areas identified for improvement. 
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2. Overview of the NLS 



The origins of the National Literacy Secretariat (NLS) lie in the 1986 Speech from the 
Throne which committed the federal government to “work with the provinces (and 
territories), the private sector and voluntary organizations to develop measures to ensure 
that Canadians have access to the literacy skills that are prerequisite for participation in 
an advanced economy.” The NLS was established in 1987. 



2.1 Program Objectives 

The NLS has two objectives: 

• To increase literacy opportunities and take-up, so people improve their literacy skills; and 

• To work towards making Canada’s social, economic and political life more accessible 
to people with weak literacy skills. 



2.2 Program Funding 

During the evaluation period (from 1997/98 to 2001/02), the NLS distributed an average 
of $29 million per year in grants and contributions (as shown in Table 1). The program 
allocates the full amount in grants from Treasury Board (TB), but converts some of the 

funds into contributions on a case-by-case basis as required by TB policy. In 2000/01, 

2 

88% of the money was distributed through grants. 



Table 1 

NLS Project Funding 


FISCAL YEAR 


TOTAL NLS PROJECT FUNDING 


1997-1998 


$29,630,851 


1998-1999 


$31,108,567 


1999-2000 


$28,679,941 


2000-2001 


$28,972,994 


2001-2002 


$26,434,006 


Source: TGN Administrative Database Technicai Report 



The NLS provides funding for projects in five project/activity areas: 

• Developing learning materials; 

• Improving coordination and information sharing; 

• Improving access to literacy programs and outreach; 



^ Source: National Literacy Secretariat (NLS) 
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• Increasing public awareness of literacy issues; and 

• Research. 

NLS project funding is distributed through two streams: 

• the national funding stream which provides support to organizations (e.g. national 
non-govemmental, provincial and territorial literacy coalitions, national literacy 
organizations, and labour organizations) in addressing literacy issues that have a national 
scope; and 

• the Federal/Provincial/Territorial (FPT) funding stream which works with the provinces 
and territories to provide funding support for literacy-based projects that address individual 
regional and local needs. 



2.3 Organization 

At the time of the evaluation, the NLS was based in the HRDC national capital 
headquarters. Its annual operating resources include $1,203,000 in salary and $1,431,850 
in non-salary dollars, which after transfers to salary Operations and Maintenance totals 
$645,501. 

2.4 Partnership 

The setting up of the NLS reflected the view of the Government of Canada that the 
advancement of literacy was too complex for any one level of government or sector to 
undertake alone. It also recognized that literacy falls within provincial jurisdiction. 
Accordingly, the NLS does not deliver programs directly but has pursued a partnership 
approach with the provinces and territories, other government departments, business and 
labour, the voluntary sector and non-govemmental organizations. 



2.5 Logic Model 

The design of the NLS can be described in terms of its logic model (see Appendix I). 
The logic model articulates the relationship between the program’s activities, the expected 
outputs and the outcomes that are expected to occur if the program is successful. To meet its 
objectives, the NLS conducts three types of activities: outreach/awareness, knowledge 
development, and project development and support. 

Various products flow from these activities. Outreach/awareness activities are aimed at 
generating promotional products, partnership development, and information-related outputs 
(e.g. websites, project databases and resource collection). Knowledge development 
activities are aimed at producing outputs such as research reports, policy documents, 
and NLS symposia. The outputs from project development and support activities include 
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advancing strategic communications with partners and recommendations regarding grants 
and contributions. 

As noted in the logic model, the short-term outcomes from the above include: 

• Enhanced strategic partnerships involved in literacy; 

• Increased evidence base; and 

• Improved information sharing and coordination; development of innovative and best 
practices; increased public awareness activities by partners; more research produced by 
partners; better trained practitioners; and more learning materials produced by partners. 

The immediate outcomes are in turn expected to lead to medium-term outcomes which 
include: 

• Stronger community networks and networking to address literacy issues; 

• Enhanced capacity of NES partners to address literacy issues; 

• Increased awareness of literacy issues by Canadians; 

• More resources toward literacy; 

• A better understanding of literacy in social and economic participation; and 

• Knowledge of best practices by practitioners, researchers and policy analysts. 

Over the long-term, NES’s activities are intended to contribute to the achievement of a 
broad range of goals including a more inclusive society, increased literacy opportunities 
for all Canadians and the removal of barriers to take-up, and the potential for full citizen 
participation in society. 

Implicit in the logic model is a decline in the direct influence of the NES as one moves 
from program activities toward longer-term outcomes. Eor example, the NES has direct 
influence over its selection of projects for funding, but has less direct influence on the 
immediate outcomes resulting from that funding. In addition, the NES is only one of 
many factors that influence on outcomes, whether they are immediate, intermediate or 
final outcomes. 
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3. Evaluation Approach 



This section outlines the issues identified for the current evaluation of the National 
Literacy Secretariat (NLS). It also highlights some of the challenges in evaluating 
partnership-styled programs, such as the NLS, the methods used to undertake the current 
evaluation, and some of the strengths and weaknesses of the evaluation approach. 



3.1 Evaluation Issues 

In accordance with the objectives of the NLS, the purpose of the evaluation is to study 
the effects of the NLS both, in terms of the opportunities for improved literacy and in 
terms or working towards making Canada’s social, economic and political life more 
accessible to people (i.e. benefits to individuals) with weak literacy skills. While the 
creation of opportunities is directly linked to the activities of the NLS, actual benefits to 
individuals may be viewed as an indirect, albeit an essential consequence of the first. 

Given these and data limitations associated with the nature of grant arrangements 
(Section 3.2), the objective of this evaluation is still to study the effects of the NLS but 
with focus on the extent to which funding resulted in products that had the potential to 
improve opportunities for literacy and, where possible, to obtain, qualitative assessments 
regarding impacts on individuals. 

Twenty-two issues were identified for the evaluation of the NLS under four broad categories: 

Program Rationale and Relevance: 

1. What is the rationale for the NLS in the current environment? To what extent has the 
role of the NLS changed since the last evaluation? 

2. Are the literacy issues as defined in 1988 still valid for the purpose of federal 
intervention today? 

Program Implementation: 

3. What is the contribution of NLS activities by funding methods, purposes and areas 
of interest? What is the incremental value of the collection of projects in the areas of: 
innovation, development and best practices, public awareness, research, the training 
of practitioners and the production of learning materials? What is the complementary 
or leveraged funding associated with those? 

4. How are the priorities established and pursued, and are there any gaps? 

5. How do the NLS activities complement and reinforce those of other partners (i.e. the 
NLS value-added). Are there critical gaps? Duplication and overlap? 
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6. What are the project selection processes of the program? Assuming the continued 
relevance and appropriateness of the objectives and priorities of the NLS, have the 
development processes for projects and the selection processes for awarding grants 
and contributions been effective in identifying project activities that are in line with 
NLS objectives and priorities? 

7. What progress has the NLS made in the promotion of plain language in the federal 
government? 

8. Progress on the new accountability framework: does regular performance reporting 
occur? Do annual reviews with grant recipients take place to determine continued 
eligibility? Does the development of an Accountability Framework enhance the 
overall policy framework for NLS, and what are the early indications and evidence? 

9. Did the program implement (or improve since the last evaluation) quality control of 
the learning materials produced or the research results? 

10. Did the program improve information sharing and co-ordination, including the 
dissemination of research results and program products to the stakeholders? 

1 1 . What criteria are used to determine when NLS intervention is no longer required 
or appropriate? 

Objective Achievement: 

12. To what extent did the program create partnerships that increase literacy opportunities 
and their take up (so that people improve their literacy skills)? 

13. To what extent did the program work toward making Canada’s social, economic and 
political life more accessible to people with weak literacy skills? 

14. What is the NLS contribution to: a better understanding of the influence of literacy in 
social and economic participation; integration of literacy considerations into related 
policy and institutional life? 

Impacts and Effects: 

15. Is there evidence of increased awareness of literacy issues by various sectors of 
society? To what extent is it attributable to the NLS? 

16. Is there evidence of an increased participation by all partners and, if yes, what has 
been the impact of this increased participation (i.e. leveraged funding)? 

17. To what extent did the NLS improve the information sharing and coordination among 
the stakeholders of the literacy sector and what are the expected impacts? 

18. Is there evidence of increased opportunities and accessibility to literacy programming 
and, if yes, what are the particular groups that have benefited (e.g. Aboriginal peoples)? 
Are the barriers to participation in literacy learning being addressed and reduced? 
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19. To what extent do research findings address the relevant issues in the literacy field, 
such as: (a) preventative measures to counteract low literacy; (b) barriers to 
participation in literacy programs; and (c) local and regional needs? To what extent 
are such research studies conducted by the NLS or funded by it? 

20. To what extent have NLS activities resulted in the development of non-traditional 
learning opportunities and innovative learning models? To what extent have the 
evaluation findings of those pilot and demonstration projects been used? 

21. To what extent can successful projects be replicated with other groups or partners and 
under what conditions? To what extent have the barriers to participation in literacy 
learning been addressed and reduced and to what extent can this be attributed to 
the NLS? 

22. Is there evidence that the NLS is responsive to emerging needs and priorities? 

3.2 Evaluation Methods 

Evaluating partnership programs such as the NLS raises a number of challenges. First, in the 
case of the NLS, the program is designed to provide project funding support to organizations 
in addressing literacy issues that are national in scope and to support literacy-based projects 
that address regional and local needs. Both of these funding streams are aimed at achieving 
the program’s objectives through groups and organizations beyond the direct influence of the 
program. As a result, in individual instances, specific final outcomes directly linked to NLS 
funding are not easy to track and measure, if at all possible. 

Second, the program funds a broad range of projects that produce diverse outputs 
(e.g. research reports, workshops, training programs) with no common metrics on which 
performance can be quantified. 

Third, timing can be a challenge. In particular, the tracking and measuring of incremental, 
cumulative change can be a balance between measuring the achievement of short-term goals 
and the attainment of a long-term vision. 

Additional challenges arise from the fact that much of NLS funding is provided through 
grants (with grants accounting for 88% of NLS funding in 2000/01). By definition, grants 
are not usually subject to audit and have only limited accountability requirements under 
which results information may be collected. Discussion of Treasury Board policy on 
grants and contributions and its impact on NLS activities is further discussed under 
Section 6.3. 

The methodology used to evaluate the NLS recognized these challenges and attempted to 
address them, for example by: 

• Emphasizing the use of multiple lines of evidence; 

• Employing both qualitative and quantitative data collection methods; 
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• Making full use of case studies to provide in-depth analysis/illustrations of program 
issues, and the potential for outputs and impacts; and 

• Using focus groups to help fill in the gaps and identify new directions. 

The main components of the evaluation approach are presented below: 



3.2. 1 Literature Review 

Prior to the evaluation, a literature review was conducted to examine methods used to 

evaluate programs with features similar to the NLS in other countries. The review 

identified the following conclusions to help guide the current evaluation of the NLS: 

• A ready-made evaluation framework is not yet available for partnership programs such 
as the NLS; 

• Other countries are placing considerable emphasis on the use of on multiple lines of 
evidence from both qualitative and quantitative sources to assess the impacts, results 
and achievements of programs that have features similar to the NLS; 

• Case studies are considered to be a key method in evaluating partnership programs and 
are used to provide concrete examples and in-depth examinations of whether and how a 
partnership program is achieving its goals; and 

• An initial document review is considered to be an important starting point and an important 
potential source of hard data for the evaluation of programs similar to the NLS. 



3.2.2 Database Review 

GMAX was the database designed for the administration of NLS grants. Because 
GMAX contains data on funded projects, it was used in the evaluation to develop an 
understanding of NLS funded activities. The database review was also used as a starting 
point for developing questions to be explored by other lines of evidence. 



3.2.3 Case Studies 

The case study analysis focused on documenting/illustrating the activities, outputs and 
results achieved by projects funded by the NLS (a summary of the 21 case studies is 
included in Appendix II). They added direct evidence of the projects undertaken and 
completed, the expected impacts and effects of NLS projects, and information regarding 
their achievements. They also added insights concerning a range of other program design 
and delivery issues. 

The case study projects were randomly selected, based on regional distribution and 
funding stream. Twenty-one case studies were completed (three funded under the 
national funding stream, and 18 funded under the FPT funding stream). 
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Each of the case studies included three research and data gathering phases: 

• A review of organizational documentation (consisting largely of a review of 
organizational websites); 

• An in-depth review of project files; and 

• Telephone interviews with project managers and key people involved in the project 
(with more than one project representative being interviewed in cases: (a) where the 
project involved more than one key player and; (b) when these individuals were still 
with the organization and available for the interview). 

The case study analysis focused on the following research areas: 

• General overview of the project; 

• An assessment of the degree to which project objectives were achieved; 

• A review of how the project was implemented, project-related monitoring and 
evaluation practices, and any obstacles that were encountered during implementation; 

• The nature of interaction with the NLS throughout the life of the project; and 

• An assessment of the longer-term impacts and effects of the project and whether the 
project was contributing to the achievement of NLS objectives.^ 

At the time of the evaluation, 9 of the 21 projects that were subject to a case study had 
not been completed on schedule (i.e., had received a deadline extension). One of the case 
study projects had been terminated prematurely and therefore was unable to achieve any 
of its objectives or intended impacts. 

A case-study review protocol was developed. The project Working Group and the 
Evaluation Instrument Validation Working Group reviewed the case-study protocol. 
The Validation Working Group is composed of literacy experts and was established with 
a mandate to review various data collection instruments to be used in the evaluation in 
order to identify any problems or issues with content, format, or presentation - 
particularly for persons with weak literacy skills. 



3.2.4 Key Informant Interviews 

Coverage included a diverse range of key informants, with an emphasis on seeking out 
corroboration of assessments of issues and outcomes. The objective of the key informant 
interviews was to gather information and data regarding the NLS’s program relevance and 
the overall impacts and effects of its funding and activities. The key informant interviews 
built on the findings from the literature review and database review undertaken at the outset 
of the evaluation. They also helped to inform the other lines of evidence. 



^ For reasons elaborated in 3.2, in individual instances, specific final outcomes directly linked to NLS funding are not 
easy to track and measure, if at all possible. Therefore, the case studies are not able to speak to impacts and effects 
on individuals. 
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At the beginning of the evaluation, employees representing all units in the NLS met in a one 
and a half day workshop to examine the foundations for the NLS evaluation. The Outcome 
Mapping Workshop focused on issues related to NLS effectiveness over the evaluation 
period. As a result of the Outcome Mapping Workshop, five key stakeholder groups 
were identified for the key informant interviews. A total of 43 key informant interviews were 
conducted with the following stakeholder groups: 

• NLS representatives: Three representatives from the NLS were interviewed regarding 
the rationale and relevance of the NLS as an organization, the effectiveness of the 
organizational objectives, the NLS partnership approach, the NLS proposal process, 
and operational issues and accountability; 

• Federal government representatives: Four federal clients of the NLS were interviewed 
(two currently work within other areas of Human Resources Development Canada 
(HRDC), and two work outside HRDC but in other federal departments); Interviewees 
were asked about rationale and relevance, including the challenges and role of the NLS, 
their views about the NLS's achievement of objectives/impacts and effects; 

• Representatives from funded organizations: Ten representatives from organizations 
that received NLS funding over the last 5 years under either of the two NLS funding 
streams were interviewed. These interviews focused on reviewing the relationship 
between the NLS and representatives of funded organizations; 

• Provincial and territorial representatives: Thirteen individuals in provincial and 
territorial governments were interviewed (these were individuals who worked with the 
NLS in the delivery of the federal-provincial/territorial funding stream, and who also 
work with the NLS to move forward issues at a national level); Interviewees were 
asked about their involvement in and their views about NLS's program implementation 
(including partnership model, program funding, leveraging, monitoring and 
evaluation), views about the NLS's achievement of objectives/impacts and effects 
(including NLS's role as a catalyst, coordination and dissemination, and future role), 
and issues related to program rationale and relevance; 

• Representatives from non-funded organizations: Thirteen individuals from 
organizations that had submitted proposals for NLS funding and were not funded were 
interviewed. These interviews were expected to provide some insight as to why some 
organizations are more successful than others at receiving NLS funding. Interviewees 
were asked questions related to program implementation (proposal submission/funding 
process and relevance of the five project areas) rationale and relevance (including 
NLS's role as a catalyst and future role), achievement of objectives/impacts and effects. 

The interview protocols included the following areas: 

• Program implementation; 

• Achievement of objectives/ impacts and effects; and 

• Program rationale and relevance. 
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Informant interviews were conducted primarily by telephone. Separate interview 
protocols were developed for each stakeholder group to tailor the information to each 
stakeholder group and to ensure that the data collection needs of the evaluation were met. 
The project Working Group and the Evaluation Instrument Validation Working Group 
reviewed the interview protocols and provided feedback. 



3.2.5 Project Survey 

A telephone survey was conducted in November 2002 and completed with 
277 representatives of NLS funded project representatives selected randomly from a list of 
627 potential contacts. The survey was designed to delve deeper into issues and further 
explore findings that emerged from the case study analysis and key informant interviews.^ 



Table 2 summarizes the project sponsor survey sample by size of organization (i.e. size of 
budget and number of employees), length of time the organization had been operating, 
NLS funding stream, and region. 



Table 2 

Some Characteristics of the Survey Sampie 


Attribute 


Organizations (%) 


Number of Empioyees (Q.34) 




One to five 


41 .5% 


Six to twenty 


38.3% 


Twenty-one to fifty 


7.6% 


Fifty-one and higher 


10.8% 


Don’t know/no answer 


1 .8% 


Budget (Q.35) 




Less than $100,000 


31.0% 


$100,000 to $500,000 


37.9% 


$500,001 to $1 million 


9.7% 


$1 million and 1 to $5 million 


8.7% 


$5 million and higher 


9.4% 


Length of time Operating (Q.33) 




Up to six years 


18.4% 


Seven to ten years 


13.7% 


Eleven to fifteen years 


20.2% 


Sixteen to twenty years 


15.2% 


Twenty years and longer 


31 .4% 



* The approximate margin of error for a sample of this size was plus or minus 6% at the 95% confidence level. 
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Table 2 (continued) 

Some Characteristics of the Survey Sampie 


Attribute 


Organizations (%) 


Funding Stream (Q.30) 




National 


28.5% 


Federal-Provincial/Territorial 


45.1% 


Not sure/don’t know 


24.2% 


No answer 


2.2% 


Region 




Atlantic 


23% 


Quebec 


28% 


Ontario 


24% 


West and Territories 


25% 



3.2.6 Focus Groups 

The focus group analysis was designed to provide program related observations 
(qualitative) and interpretations on the impacts and effects of the program. A decision 
was made to conduct the focus groups with stakeholders that did not necessarily have a 
direct relationship with the NLS, but who might have useful contributions to make. Thus, 
one focus group was held in Toronto with literacy experts/academics, and another focus 
group was held in Ottawa with like organizations facing similar challenges as the NLS in 
the areas of funding projects, partnership activity, performance measurement, etc. 

The focus group with literacy experts/academics included representatives from the following 
organizations: the University of Toronto Adult Education and Counseling Psychology 
department; the YMCA of Greater Toronto; Clear Language and Design (a division of the 
Toronto East End Literacy Project); Preparatory Training Branch (a non-profit organization 
that has helped over 8,000 participants develop solid literacy and basic skills); the Metro 
Toronto Movement for Literacy; and two academics (one associated with the University of 
Toronto and one from St. Erancis Xavier University in Nova Scotia). 

Participants in the focus group of l ik e organizations included representatives from the Justice 
Grants and Contributions Eund (Department of Justice); the National Secretariat on 
Homelessness (HRDC); the Eederal Partners in Technology Transfer Program, National 
Research Council; the Petal Alcohol Syndrome Division (Health Canada); and the United 
Way of Ottawa. 
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3.3 Strengths and Limitations of the 
Evaluation Approach 

The evaluation methodology developed to evaluate the NLS had the following strengths: 

• Took a comprehensive approach and involved a wide range of stakeholders related to 
the NLS and the literacy field; 

• Emphasized the use of multiple lines of evidence to allow the findings from one approach 
to substantiate/corroborate and inform findings from other lines of evidence; 

• Interviewed key informants with multiple perspectives on evaluation questions. 
For example, representatives from provincial and territorial governments were inclined to 
view the issue of literacy, and the role of the NLS, not simply from a program 
effectiveness perspective but through the lens of federal/provincial relations; 

• Conducted case studies analysis to provide more concrete analysis/illustrations of 
program issues, outputs and impacts (where such information was available). 

Although the evaluation approach developed for the NLS recognized and attempted to 

address the challenges in evaluating this type of partnership-style program, the following 

limitations should be noted: 

• Information to confirm specific outcomes of the NLS was often not available and the 
project representatives interviewed as part of the case study analysis often had no views on 
key issues, or were simply not very well informed. For example, very few of 
the interviewed project representatives were aware of the objectives of the NLS prior to the 
interview, and only one-third were able to provide in-depth commentary on the subject; 

• There were challenges encountered in using GMAX for analytical purposes. 
For example, coding limitations only allowed a project to be coded against one activity 
even though it may have met the goals of multiple activities, thus limiting the 
usefulness of GMAX in providing an in-depth perspective of NLS activities. 
(The nature of the data being collected and how it is stored should be examined to 
ensure that appropriate data is available for a future evaluation of the NFS.); 

• Representatives from non-funded organizations (those who were unsuccessful in 
getting NLS funding) had little knowledge of either the literacy issues or the Secretariat 
itself, and were often unwilling to participate in the evaluation. The majority of 
non-funded organizations that were interviewed tended to be small, with limited 
resources and high turnover of personnel associated with their NLS proposal. As a result, 
finding appropriate people to interview from the non-funded organizations was a major 
challenge (i.e. forty-nine organizations were contacted in order to complete thirteen 
interviews). It should be noted that difficulties in surveying refused/non-funded 
organizations are not unusual. Organizations are less likely to participate in studies of the 
program that rejected their proposals than organizations that received program funding. 
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4. Impacts and Effects 



This section begins with an overview of National Literacy Secretariat (NLS) funded projects 
and then considers the impacts and effects of NLS funding by examining the incremental 
effects of NLS funding, the role of the NLS as a catalyst, program impacts on various groups, 
leveraging and the issue of sustainability. 



4.1 Overview of NLS Projects 

NLS funding is broadly dispersed across many organizations. 

The NLS distributed an average of $29 million per year in grants and contributions 
during the evaluation period. 

NLS funding through the two funding streams is broadly dispersed. From 1997 to 2002, 
2,043 organizations received funding for 2,853 projects. Four-fifths (80%) of these 
organizations had just one project over the five-year period, and accounted for 57% of all 
NLS projects during that time. The maximum number of projects that any 
one organization received funding for was nine, and almost all received funding for fewer 
than four projects. The largest recipients of NLS funds were literacy organizations. 

Just over half of all projects funded by NLS received less than $25,000. 

In terms of the size of the funding provided by the NLS: 

• Just over 50% of all projects funded by NLS received less than $25,000; 

• 33% were funded between $25,000 and $75,000; 

• 13% were funded between $75,000 and $200,000; and 

• The remaining 3% were projects funded at more than $200,000. 

From 1997 to 2002, there was a slight upward shift in the funding of projects; as the 
proportion receiving less than $25,000 declined, while those receiving between $25,000 
and $75,000 increased. 

Projects focused on learning materials consistently accounted for the greatest 
proportion of projects funded during the evaluation period. 

In order to be eligible for project funding through the NLS, proposals must fall under at 
least one of the following five project/activity areas: developing learning materials, 
improving coordination and information sharing, improving access to literacy programs 
and outreach, increasing public awareness of literacy issues, and research. 
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A review of the project areas funded by NLS over the past 5 years (shown in Figure 1) 

indicates: 

• Projects focusing on learning materials consistently made up the greatest proportion of 
all projects funded during the five-year period, although there was considerable 
fluctuation in the share of projects they accounted for and a declining trend over the 
five-year period; 

• Research and public awareness projects also experienced fluctuations in the share of 
projects they accounted for over the five-year period, with a slight upward trend in the 
case of research projects; and 

• The project categories of coordination and information, and access and outreach, were 
fairly steady over the five-year period. 

The processes used by the NLS to select projects for funding are examined in Section 6. 




4.2 Incremental Effects 

Incrementality, in program evaluation terms, refers to the difference a program made net 
of what would have happened in the absence of program funding. In the context of the 
NLS, and the projects funded, project incrementality is equal to the total projects funded 
less those projects that would have proceeded in the absence of NLS support. Similarly, 
the incrementality of the funds leveraged by project sponsors from partners is equal to 
total funds leveraged less those funds that could have been invested in NLS-type literacy 
activities if the program had not existed. 
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The available evidence suggests that only a small percent (less than 4%) of the 
projects would have proceeded at all without NLS funding, but this is difficult to 
confirm. 

Less than 4% of the organizations responding to the survey on funded projects indicated 
that their project would have proceeded without NLS funding. 

Key informants from organizations with funded projects indicated that NLS funding is 
viewed as critical in supporting their work at the community level. Also, many of the 
provincial and territorial key informants indicated that, without NLS funding, provinces 
and territories would not have had the resources to support projects in the area of 
workplace and family literacy. 

It should be noted, however, that it is difficult to draw a final conclusion regarding 
incremental effects because it is difficult to separate the effects of NLS funding from 
other sources of support for NLS (as discussed in Sections 4.3 and 4.5, and in Section 5). 
However, the lack of evidence to support evaluation findings on incremental effects 
points to the importance of ensuring realistic program safeguards are in place to protect 
against project overlap and duplication to the extent possible. The issue of program 
safeguards is discussed further in Section 6.5. 



4.3 Role of NLS as a Catalyst 

Increasing Awareness 

The NLS has increased awareness of literacy issues. 

The case study analysis indicated that the NLS has contributed to increased awareness and 
understanding of the importance of literacy to social inclusion and ability to participate fully 
in society. 

All thirteen of the provincial and territorial informants credited the NLS with raising the 
profile of issues related to literacy. 

All ten of the key informants from organizations with funded projects indicated that the 
NLS has been successful at increasing the awareness about literacy issues in their 
organizations and among their contacts throughout Canada and for more broadly defining 
what literacy means to society. 

Developing Partnerships 

The available evidence indicates that the NLS contributed to bringing people 
together and developing partnerships in the area of literacy. 

Almost all of the case study projects reported some form of consultation/collaboration 
within the literacy community and/or learners within the community to determine needs, 
establish best practices, or engage in project activities, such as developing or testing 
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learning tools. In almost all cases, projects benefited from collaborating with various 
community partners. 



Organization: Institut canadien d’education des adultes 

Project: Promotion, sensibilisation et formation aupres d’ organismes canadiens francophones de 
rOuest et du Nord - Nos competences fortes (NCF)2 

Location: Quebec 

NCF is a set of tools that was designed to allow French-speaking adults who may or may not 
have efficient writing skills or educational backgrounds to assess their own competencies and 
become more productive individuals within their communities. 

This project began in 1998, at the general meeting of the Canadian Literacy Federation in French, 
held in Halifax. The francophone population of Western Canada wanted to know how to obtain 
funding to help literacy initiatives within their respective communities. It took approximately 
three years of talks and negotiations with the four western provinces to make substantial progress. 
Throughout this process, NLS representatives facilitated the discussion, which helped to ensure 
that the project was finally launched. It was felt that, had it not been for NFS’s role in bringing 
together all involved parties, the project would never have gone forward. 



Organization: Nunavut Literacy Council 

Project: Literacy & Community Development Workshops 

Location: Nunavut 

The Nunavut Literacy Council implemented a project aimed at raising awareness and 
knowledge of community capacity building and literacy development among various groups 
such as Literacy Council Board members, educators, Inuit organizations, literacy practitioners 
and government officials by delivering literacy-related workshops. This has promoted 
partnerships within communities in the territory and the organization has subsequently been 
overwhelmed by the demand for its services. 



Key informants from organizations with funded projects credited the NLS with creating and 
supporting partnerships throughout the literacy community. All thirteen of the provincial and 
territorial key informants also credited the NLS with bringing people together. 

The focus group of literacy experts credited the NLS for being creative in bringing together 
people who do literacy work with academic researchers and encouraging a concerted focus 
on literacy issues. 

Advancing the Literacy Movement 

The evidence indicates that the NLS has been a catalyst for literacy initiatives. 

The case study analysis indicates that the NLS has advanced the literacy movement, 
for example by encouraging practitioners to think of new ways to improve their reach, 
creating visibility for organizations and issues, bringing professional expertise to the area, 
creating new recruitment opportunities for under-resourced organizations, and increasing the 
overall level of sensitivity to literacy issues. A majority of the case studies indicated that 
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funding from the NLS has also allowed organizations to focus on key problem areas and has 
improved their ability to deliver successful literacy programs within the community. 



Organization: Centre for Education and Work 
Project: Articulating Workplace Education 

Location: Manitoba 

The objective of this project was to design and develop a pilot system for the articulation of 
essential skills work completed in the workplace. This project positions the workplace as a 
meaningful place of learning for adults returning to education. When completed, the project 
will have developed systems so that adult learners can receive formal recognition for their 
workplace learning efforts in various educational and employment settings. 



Key informant interviews corroborated the findings of the case studies, with specific 
groups expanding upon certain themes. All of the thirteen provincial and territorial key 
informants credited the NLS with acting as a catalyst by encouraging them to provide 
budgetary resources for literacy programs, bringing people together, promoting research 
and raising the profile of literacy as a national issue. 

The focus group of literacy experts credited the NLS with providing what it called the 
‘space’ for practitioners, educators, academics and others to think, reflect, and collaborate 
in order to move literacy forward. The focus group also credited the NLS for supporting 
the National Adult Literacy Database. 

Client Satisfaction 

As shown in Table 3, 70% of the organizations surveyed indicated that they were either 
satisfied or very satisfied with NLS services. Looking at the five activity /project areas of 
the NLS, the surveyed organizations are most satisfied with the performance of the NLS 
in the area of “encouraging the development of learning materials and methods” (81%) 
and are least satisfied with NLS performance in the area of “access and outreach in order 
to increase participation in literacy programs” (59%). 
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Table 3 

Satisfaction with NLS Performance 





Satisfaction 




Very 

dissatisfied 


Dissatisfied 


Neutral 


Satisfied 


Very 

satisfied 


Don’t 
know/ no 
answer 


Overall (Q.10) 


1 .8% 


8.7% 


15.9% 


46.9% 


23.5% 


3.2% 


Learning materials 
and Methods 
(Q.11a) 


0.4% 


2.9% 


1 1 .2% 


50.2% 


31% 


4.3% 


Access and 
Outreach (0.11b) 


1 .8% 


9.7% 


24.5% 


40.4% 


19.1% 


4.3% 


Promotion and 
Public Awareness 
(0.11c) 


0.4% 


9.7% 


18.1% 


46.9% 


18.4% 


6.5% 


Coordination and 
Information sharing 
(O.lld) 


0.7% 


10.1% 


19.5% 


41 .2% 


24.9% 


3.6% 


Applied research 
(0.1 1e) 


0.7% 


4.3% 


19.5% 


46.9% 


19.9% 


8.7% 


Source: Survey of funded projects 



Overall satisfaction with the NLS is higher among organizations with bigger budgets. 
Organizations with annual budgets of $500,000 or above are more apt to be satisfied 
(82%) with the overall support of the NLS, compared to those with budgets below 
$500,000 (66%). One possible explanation is that the larger organizations usually work 
more closely with the NLS under the national funding stream. 

Overall satisfaction with the NLS is highest in Ontario (35% are very satisfied) and 
Western Canada (39% are very satisfied), and lowest in Quebec (5% are very satisfied). 
The low satisfaction rate in Quebec may be a function of the partnership relationship 
which is very limited in this province. Representatives from Ontario and Western 
Canada are also more satisfied with the performance of the NLS in each of the five 
activity areas, compared to other parts of Canada. 



4.4 Impacts on Canadian Groups 

There is evidence that NLS funding has improved literacy opportunities for a range 
of groups including people with low incomes and people with disabilities. 

Two-thirds of the case study organizations noted that the NLS has made an important 
impact in areas such as family and workplace literacy. 

The majority of the provincial and territorial key informants supported the evidence from the 
case study analysis that the NLS is improving the economic and social life of people with 
weak literacy skills through its expansion of literacy to target workplace and family literacy. 
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The Key Informants noted that the NLS has improved the level of service to workplace 
literacy students by elevating workplace literacy and raising the professionalism of 
workplace literacy providers/practitioners. Prior to the 1990’s, community educators 
were involved in workplace education, but now as a result of the efforts of the NLS, 
as well as those of its partners, the teaching of adult and workplace literacy is viewed as 
being “professionalized.” 

The project survey asked the project representatives if their project improved literacy 
opportunities for any of a range of groups within the Canadian population. As indicated 
in Table 4, the project representatives were most likely to report that their projects 
improved literacy opportunities for people with low incomes (83%), for 
practitioners/tutors (69%), for people with disabilities (55%), first generation Canadians 
(40%), Aboriginal people (36%), seniors (35%) and, parolees and inmates (18%). 







Table 4 








Impact of NLS Funded Project on Literacy Opportunities 






Impact 












Not 


Don’t know/ 


Population Group 


Not at all 


A Little 


Some 


A Lot 


applicable 


no answer 


People with Disabilities 
(Q.31a) 


19.5% 


13.0% 


31.8% 


23.8% 


6.1% 


5.8% 


Senior Citizens (Q.31b) 


36.1% 


18.4% 


23.8% 


1 1 .2% 


6.5% 


4.0% 


Aboriginal People (Q.31c) 


33.2% 


14.1% 


23.1% 


13% 


12.6% 


4.0% 


Francophones (Q.31d) 


32.9% 


8.3% 


14.1% 


33.9% 


7.9% 


2.9% 


First Generation 
Canadians (Q.31e) 


27.4% 


17.7% 


23.5% 


16.6% 


9.0% 


5.8% 


Parolees/ Inmates (Q.31f) 


50.9% 


12.6% 


10.8% 


6.9% 


12.6% 


6.1% 


Practitioners/ Tutors 
(Q.31g) 


12.6% 


9.7% 


26% 


43.3% 


6.1% 


2.2% 


People with Low Incomes 
(Q.31h) 


2.2% 


6.5% 


24.2% 


59.6% 


4.3% 


3.2% 


Sources: Survey of funded projects 



To examine the potential of wider impacts, the project survey also asked whether 
the organization’s involvement with the NLS had contributed to the overall success of the 
organization. Most of the project representatives (88%) indicated that being involved 
with the NLS had contributed in this fashion. 

All of the key informants from organizations with funded projects indicated that funding 
from the NLS allowed their organizations to focus on key problem areas and improve the 
ability to deliver successful literacy programs within the community. 

NLS’ impact is discussed further in Section 5.2 with regards to the program’s achievement of 
its objectives. 
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4.5 Leveraging 

In the context of the NLS evaluation, there are two instances in which leveraging might 
occur - the project leveraging of financial and/or in-kind resources from partners and the 
leveraging of knowledge acquired by the outputs of the program (i.e. project leveraging). 
Leveraging of resources refers to the use of funds to generate additional funds or in-kind 
resources (such as computers, office supplies, volunteers, etc.). Project leveraging refers 
to the application of projects to build on the knowledge generated by previous projects or 
to assist with the development of other projects. 

There is some evidence that NLS funding is leveraging funding and in-kind types of 
assistance from other sources in support of literacy-based initiatives. 

Almost half of the case study organizations indicated that initial NLS funding has 
assisted in leveraging funding, as well as other support materials from partners. 

About half of the funded organizations included in the project survey indicated that 
they were able to use their NLS funding to leverage other funds, and about two-thirds 
indicated that they were able to leverage other types of support. Organizations with higher 
annual budgets (at least $500,000) appear more likely than smaller organizations to 
leverage additional funding and other resources. Organizations funded under the national 
funding stream also appear more likely to leverage other funds and resources. 



In Saskatchewan, the NLS is credited with initiating funding for family literacy development 
in the province. As a result of NLS seed funding and other developments in the province, 
the family literacy initiative now has a budget of $300,000 per year. 







This conclusion is consistent with key informant interviews. All thirteen provincial and 
territorial key informants credited the NLS with encouraging them to provide money for 
literacy programs. The FPT funding stream was originally developed to encourage 
provinces to begin to fund literacy initiatives, and in a majority of the larger provinces 
the provincial governments are now contributing more funds than required under the 
match-funding program. 

It has to be recognized that the available evaluation information collected on leveraging 
is impressionistic at best. Leveraging estimates have to be assessed in the context of the 
program’s incremental impacts and here the evidence is ambiguous (see Section 4.2., 
Incremental Effects). Furthermore, even if leveraging is taking place, the data presented 
in Table 5 below point to the conclusion that NLS project funding as a proportion of total 
project budgets is increasing over time (i.e., NLS funding is playing a larger (percentage) 
role in total project funding). 
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Table 5 

Comparison of NLS Project Funding to Project Budgets 



Project Budget 



Fiscal Year 


Total NLS 
Project Funding 


Total Project 
Budgets 


NLS Project Funding 
as a Percent of 
Total Project Budgets 


1997-1998 


$29,630,851 


$66,078,274 


44.8 


1998-1999 


$31,108,567 


$64,620,382 


48.1 


1999-2000 


$28,679,941 


$54,720,518 


51.7 


2000-2001 


$28,972,994 


$57,139,598 


50.8 


2001-2002 


$26,434,006 


$49,252,370 


53.5 



Source: TGN Administrative Database Technicai Report 



The available evidence that NLS funding is leveraging funding from other sources 
indicates the need for the NLS to be able to demonstrate that it has program 
safeguards in place to ensure that NLS funded projects and the leveraged funds 
are incremental. 

Given that NLS funded projects have the potential for leveraging funding and in- ki nd support 
from other sources, there is a need for the NLS to be able to demonstrate that it has program 
safeguards in place to reduce the risk that NLS funded projects would have gone ahead and 
leveraging would have occurred in the absence of NLS support. The issue of program 
safeguards is discussed further in Section 6.5. 



4.6 Sustainability 

The evidence indicates that more than half of the NLS funded projects continued 
after NLS funding expired. 

Almost half of the case study organizations indicated that NLS funding has the potential 
for moving programs to become autonomous or self-sustaining. Only one organization 
indicated that, although its program was successful, it was unable to continue on with its 
program after the NLS ceased to fund the project (Comite Alpha Papineau: Project - 
L’Alphabetisation par les pairs). 

Half of the project representatives included in the project survey indicated that their project 
continued after the NLS funding expired, 31% indicated that their projects did not continue, 
and 13% indicated that this was not applicable (i.e. the project was not intended to continue 
after the funding period). This evidence suggests that over half the projects that received NLS 
funding during the evaluation period became sustainable, while almost one-third were unable 
to continue. 

More than half of the key informants from organizations with funded projects identified 
an ability to sustain the projects after NLS finished funding the project. 
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The issue of sustainability is also closely linked to the findings above on the need for a 
clearer perspective on the program’s incremental effects. The fact that in nearly one-third 
of the cases examined sustainability was not maintained is important corroborative 
evidence that without the support of the federal initiative the projects would possibly not 
have proceeded at all in these cases. On the other hand, for those projects that did 
continue, it is not clear at present what factors contributed to this continuation and the 
role of initial NLS funding in this context. 

The issue of sustainability is discussed further in Section 7. 
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5 . Achievement of Objectives 



This section looks at the extent to which the National Literacy Secretariat (NLS) is 
achieving its two overall objectives: 

• To increase literacy opportunities and take-up, so people improve their literacy skills; 
and 

• To help make Canada’s social, economic and political life more accessible to people 
with weak literacy skills. 



5.1 Awareness of NLS Objectives 

Many of the organizations receiving NLS funding are not well informed of the 
overall objectives of the NLS. 

Very few of the case-study organizations were aware of the overall objectives of the NLS 
prior to being informed of these objectives as part of the case study analysis. 

Some key informants were also unaware of the overall objectives of the NLS. 

On the other hand, the focus group of literacy experts was aware of the two objectives of 
the NLS and was very supportive of them. 



5.2 Evidence of Achieving NLS Objectives 

There is some evidence to suggest that NLS projects are achieving their stated 
objectives and contributing to the broader objectives of the NLS, although it is 
unclear to what extent the NLS has been successful in meeting its stated objectives 
in part due to issues of measurement and attribution. 

Case Study Evidence 

Once the representatives of the case study organizations were informed of the overall 
objectives of the NLS as part of the case study analysis, they all indicated that they felt 
that NLS was achieving them, although only one-third of the case study organizations 
were able to provide in-depth commentary on this subject. 

Regarding access to literacy opportunities, for example, one of the case study organizations 
indicated that participation in literacy-oriented programs had been minimal in the past 
because the programs were not provided free of charge within its community and were 
narrowly targeted (only to select groups). Through NLS project funding, however, 
participation in their literacy programs has increased. 
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Organization: The Canadian Mental Health Association 
Project: Barriers to Literacy 
Location: Prince Edward Island 

This project involved hiring a literacy coordinator to determine whether greater coordination 
could improve participation in adult literacy programs. The project revealed that it was a 
misconception that adults were not participating in literacy programs due to the stigma 
attached to having low literacy skills. The project demonstrated that there was a lack of 
coordination, and that adult learners did not know what programs were available or how to 
access them. The project also helped 20 people take the first steps toward improving their 
literacy skills by accessing the services available to them. 




— mmm '■ 



Organization: Institut canadien d’education des adultes ■ 

Project: Promotion, sensibilisation et formation aupres d’organismes canadiens francophones de I 
r Quest et du Nord - Nos competences fortes ( NCF)2 I 

Location: Quebec I 

In Quebec, a set of tools called nos competences fortes (Our Strong Competencies) was designed 
to allow French-speaking adults with literacy needs to assess their own competencies so that they 
could find ways to leverage their strengths to become more productive individuals within their 
communities. This project aimed to promote these tools and provide training to practitioners so 
they could be applied by Western and Northern francophone organizations. This project I 
contributed to making “Canada’s social, economic and political life more accessible to people with 
weak literacy skills”. , 



Project Sponsor Survey Evidence 

As discussed in Section 4.4, the project survey indicated that projects funded by the NLS 
had improved literacy opportunities for a number of groups, including people with low 
incomes, practitioners/tutors, people with disabilities, and first generation Canadians. 

Key Informant Evidence 

Key informants also provided some support for the conclusions that NLS projects were 
contributing to the overall objectives of the NLS. For example, more than half of the 
13 provincial and territorial key informants indicated that the projects and innovative 
activities funded by NLS are leading to increased opportunities for individuals at the 
community level to improve their literacy skills. 

Regarding the NLS objective of making Canada’s social, economical and political life 
more accessible to people with weak literacy skills, the provincial and territorial key 
informants emphasized that literacy is a long-term issue involving generational change. 
The majority also indicated that the NLS is improving the economic and social life of 
people with weak literacy skills through its expansion of literacy to target workplace and 
family literacy. 
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The following findings from key informant interviews provide perceptions of how the 
NLS is achieving its objectives of building capacity, awareness and access through its 
five key program activities. These findings also identify the need for better sharing of 
information (discussed further in sections 6.1 and 6.4), and for program safeguards to 
ensure quality control in research and to reduce the potential for project duplication 
(discussed further in Section 6.5): 

• Development of learning materials and methods: The NLS is viewed by the majority 
of the key informants as successfully developing and sharing learning materials for use 
throughout the literacy community; 

• Improved access and outreach to increase uptake of improved literacy: The NLS is 
viewed as funding critical projects that are designed to support community-based outreach; 

• Promotion and public awareness: The NLS is credited with having increased the 
awareness of literacy issues; 

• Coordination and information sharing: The NLS is viewed by all as being helpful in 
this area. The National Adult Literacy Database that was established by the NLS is 
viewed as one of the most effective outputs from the NLS’s activities, with one of the 
provincial representatives calling it a “virtual gold mine of information.” The NLS was 
also given positive feedback on the electronic communications systems that it funds. 
The majority indicated that better systems were needed to track projects that have been 
funded, either through the NLS or the provinces and territories. In particular, there is a 
need to be able to identify at the proposal stage whether a similar project has been 
funded elsewhere in Canada to ensure that the funded projects either build on the 
existing literature/findings or take a new direction. The majority of the key informants 
indicated that they were unaware of the wide variety of projects funded by the NLS and 
felt that they would benefit if the NLS could better communicate on the funded 
projects and disseminate outputs that have passed through a peer review process; 

• Applied Literacy Research: NLS funds three major types of research: (1) large-scale, 
national research projects, some of which are undertaken jointly with SSHRC and 
Statistics Canada; (2) research funded under the federal-provincial stream and; (3) 
needs assessment of literacy issues and needs in local communities. The majority of 
the key informants noted that the NLS has added to the knowledge base by funding 
applied research related to literacy. The International Adult Learning and S ki lls 
Survey (lALSS) is viewed as a landmark piece of work for describing the situation in 
Canada in comparison with other countries in the world. It was noted that the NLS also 
funds other less formal research, as well as applied research from the literacy 
community. About half of provincial and territorial key informants indicated that the key 
strengths of the NLS relate to research, data collection and dissemination. The majority 
of the key informants recommended that the NLS provide improved quality assurance 
(e.g., peer review) of all research. It was generally recognized that the provinces are not 
resourced for quality control, but that there is a need within the literacy community for 
this activity. 
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The majority of the key informants from organizations with funded projects indicated that 
the projects funded by NLS were more likely to support the first objective of the NLS. 

Some differences were noted across the country. Key informants from Ontario generally 
indicated that program objectives are far removed from what happens at a project or 
community level. Those from other parts of Canada, particularly those located in Quebec 
and on the East Coast, feel that NLS projects stimulate people to continue to learn, which 
ultimately increases their ability to participate more fully in society. 

The key informants from organizations with funded projects credited the NLS with helping 
to disseminate research findings across Canada; approximately one-quarter of the key 
informants noted that the NLS often makes presentations and conducts briefing sessions on 
workplace education and tools for assessment. These tools are now being applied in virtually 
every province or territory and in private sector and labour organizations. 

It should be noted, however, that some of the key informants felt that it was difficult to 
determine the extent to which the NLS is meeting its stated objectives: 

• A few of the key informants from organizations with funded projects noted that overall 
objectives of the NLS are very similar to those of most literacy-based organizations, 
which makes direct attribution to the NLS difficult; and 

• In relation to the objective centered on increased literacy opportunities and take-up so 
people improve their literacy skills, a majority of the provincial and territorial key 
informants indicated that this falls within provincial jurisdiction (i.e. direct programming), 
making it difficult to make attribution for success to the NLS. The majority of the 
provincial and territorial key informants found that it was even more difficult to make 
attribution for success related to the objective of making Canada ’s social, economic and 
political life more accessible to people with weak literacy skills. 

Focus Group Evidence 

The focus group of literacy experts maintained that the objectives of the NLS were 
important enough to be pursued and that the NLS is making a positive impact on the 
country, and that the provinces and territories benefit from the federal government’s 
support in this way. 
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6. Program Implementation 



This section examines the National Literacy Secretariat (NLS) partnership model and the 
application process. It also examines the issue of monitoring and evaluation, and takes a 
closer look at the issues of dissemination of information, and overlap and duplication. 



6.1 The NLS Partnership Model 

The NLS pursues a partnership approach with the provinces, territories, voluntary groups 
and associations, and business and labour organizations. Rather than being involved in 
direct program delivery, the NLS works with and through its partners to enhance literacy 
in Canada. 

The partnership model followed by the NLS for the Federal-provincial-territorial 
(FPT) funding stream has been functioning well. 

The FPT funding stream supports literacy-based projects that address regional and local 
needs. The FPT stream involves an annual call for proposals through which organizations 
tender submissions directly to the province or territory. 

Under the FPT funding stream, the NLS has structured a working partnership designed to fit 
with the specific needs of each province and territory. For example, some provinces, such as 
New Bmnswick, have a matched funding program agreement with the NLS. Under these 
agreements, the province matches the program funding provided by the NLS and projects are 
therefore funded jointly. In other provinces, such as Manitoba, the province funds direct 
programming and the NLS funds projects related to research, development and innovation. 
In provinces such as Ontario and Alberta, the NLS and the province work together to fund 
specific areas of literacy; however, the provinces have a much larger funding envelope for 
literacy activities. In provinces such as Quebec, the province manages the funding envelope 
and makes recommendations to the NLS for project financing. 

The majority of provincial and territorial key informants indicated that they work with 
the NLS on an annual cycle to establish funding priorities and to develop and extend calls 
for proposals. Four provinces felt that NLS’s priorities were not always consistent with 
those of their provincial government. 

In each province and territory, there is a joint review process, which varies from province 
to province to permit flexibility and to meet the needs of each jurisdiction. 

All provincial and territorial key informants indicated that they are satisfied with the 
partnership model that is used under the FPT funding stream. There is agreement among all 
those consulted that the working relations between the NLS and their provincial and 
territorial counterparts are positive and productive. The majority believe that the processes 
related to this program are effective and the communications are good. 
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Communications with partners in relation to the selection and announcement of 
projects funded through the national funding stream was identified as an area 
for improvement. 

The national funding stream provides support to organizations in addressing literacy issues 
that have a national scope. Proposals for this stream are accepted directly by the NLS on an 
ongoing basis and reviewed by a national committee in the NLS at regular intervals. 

The majority of the provincial and territorial key informants expressed dissatisfaction with 
how the national funding stream functions. While provincial and territorial representatives 
recognize that the national funding stream falls within federal jurisdiction, they felt that 
decisions made by the NLS under this stream can have an impact at the provincial level. 
All provinces/territories felt that they were not adequately informed of funding and project 
decisions under the national stream. In one example, the province of Saskatchewan noted that 
the NLS had funded an organization within the province for a work place literacy project. 
The province was also funding the same organization, but had minimal input into the NLS 
decision. Thus the province and the NLS were not viewed as working in a complementary 
fashion with possible implications for problems of overlap and duplication. For further 
discussion of overlap and duplication see section 6.5. 

The degree of ongoing interaction with the NLS varies by project. 

The case study analysis found varying degrees of ongoing interaction between the funded 
organizations and the NLS. Four organizations that received funding through the FPT 
stream reported strong working relationships with the NLS. Of the remaining fourteen 
FPT organizations from the FPR funding stream, ten reported working closely with the 
province and four progressed independently throughout the proposal process and project 
implementation. The three nationally funded organizations funded through the national 
funding stream viewed the NLS as a valuable partner who provided ongoing feedback 
and support throughout their projects. 

Half of the case study organizations did not consider the NLS to be an active partner. 
These organizations received funding through the FPT funding stream and therefore had 
more interaction with the province than with the NLS throughout the proposal and 
implementation stages of their projects. One of the case study organizations stated that 
the need to interact at the provincial level inhibits the development of what they believed 
would be a valuable relationship with the NLS. 



Organization: Open Doors Adult Literacy Program 
Project: Journaling: A Resource Guide 

Location: Manitoba 

As a project funded under the federal-provincial/territorial stream, Open Doors did not work 
with the NLS to develop its proposal, but rather worked with the provincial representative. 
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Organization: Ontario Literacy Coalition 
Project: Raising the Profile of Workplace Literacy 

Location: Ontario 

This project was established to develop a comprehensive marketing strategy to promote workplace 
literacy to the private sector. It involved the coordination of five regional projects focusing on 
marketing workplace literacy. This is a partnership with QUILL Network, Literacy Network of 
Durham Region, Mohawk College, CGPC and Metro Labour Education Centre. 

The NLS proposal process is viewed as working well, primarily because it is a cooperative 
approach where all those involved work together to ensure that funds are fully leveraged to 
support a project. In most cases, the OLC develops concepts for projects based upon identified 
needs and then consults with the NLS for their views before drafting a full proposal. It can 
take a number of months to ensure broad consultation on both sides. Throughout the 
proposal/approval process, it was noted that for the most part, adequate and straightforward 
guidance is provided by NLS. 

The NLS is viewed as being flexible when making funding decisions and innovative, a quality 
that has helped the literacy field to grow over time. 



A majority of the key informants from the funded projects believe that the partnership 
approach employed by the NLS is key to the whole process, especially given that education 
and training is a provincial responsibility. 



On Partnerships 

‘Partnerships must be considered in terms of continual relationships. You have to be able to • 
clearly define your expectations and the expectations of your partners. Often you come in as a 
funder and partner, and they see you as a funder only’. I 

Focus group of like organization, i 



The focus group of literacy experts was very supportive of the NLS partnership efforts. 
They credited the NLS with providing the vision and support for the notion that those 
working in literacy need to share with each other through the creation of cross-sectoral and 
multi-dimensional projects. The focus group of like organizations discussed their experiences 
with partnership arrangements, noting that it is important that the partners be involved in 
setting priorities that complement a program’s strategic objectives. 



6.2 The NLS Application Process 

There appears to be some confusion surrounding the role of the NLS in developing 
proposals for the FPT funding stream. 

Organizations that qualify for the FPT funding stream generally consult with their provincial/ 
territorial government office regarding proposal requirements, deadlines, application forms, 
and assessment procedures. Proposals are therefore submitted to these offices, which then 
work with the NLS to select funding recipients. While the NLS is available to assist and 



Summative Evaluation of the National Literacy Secretariat 





support organizations throughout the proposal development phase, knowledge of the 
availability of NLS assistance among organizations that sought funding under the FPT stream 
was low. 

Many of the successful applicants indicated that the application process and 
guidelines were clear, although some suggested that smaller organizations could 
benefit from more support. 

The three case study organizations funded through the national funding stream and three of 
the eighteen case study organizations funded through the FPT stream (and who worked 
closely with the NLS prior to submitting their project proposals) reported a clear 
understanding of the NLS requirements and priorities. Representatives in these organizations 
felt that they were working in partnership with the NLS and valued its support and advice. 

Three-quarters of the case study organizations found the proposal guidelines clear. 

Two of the case study organizations were not satisfied with the proposal acceptance 
notification process. One organization felt that five months was too long to wait. Another 
was disappointed that it received word of its approval for project funding second-hand 
through the local media, rather than directly from the NLS or the provincial/territorial 
representative. The remaining organizations had no comment or were generally pleased 
with the notification process. 

Unsuccessful applicants were much more critical of the application and notification 
process. 

Key informants from the non-funded organizations were unhappy with the proposal 
process. All thirteen remained emphatic about the merits of their proposed idea and were 
unsure as to why their proposal was not accepted for NLS funding. All were also unhappy 
with how they were notified that their proposal was rejected. All had received a letter stating 
that they would not receive funding and wishing them good luck in their future endeavors. 
All expressed dismay that there was no constmctive feedback, and three tried to seek out 
feedback (i.e. two sought feedback from their provincial coalition and one from the NLS). 
The one who sought feedback from the NLS was unable to obtain any feedback. 

Approximately half of the key informants from the non-funded organizations commented 
that they would have benefited from further instructions and guidelines for completing 
their proposal. 

The NLS needs to put in place program safeguards to ensure the quality of its 
research projects and project outputs. 

About half of the provincial and territorial key informants and the majority of other key 
informants indicated that there was a need for the NLS to have some safeguards in place 
to ensure the quality of research and outputs from projects funded by the NLS. A peer 
review process was identified as a way to help ensure that NLS funded research projects 
and outputs meet the standards of the literacy community. 
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6.3 Monitoring and Assessment 

Due to the nature of the grants process, as outlined in the Treasury Board Policy on Transfer 
Payments, the implementation of accountability checks and balances during the life of the 
project is limited to stringent proposal review and approval. By definition, a grant is not 
subject to being accounted for or audited. Proposals must only meet eligibility criteria or 
preconditions for entitlement to be verified. In keeping with this requirement, the NLS’ 
accountability process of grants focuses on the development of the proposal and the capacity 
of the organization to deliver on its project commitments. The NLS further requires that each 
project submit a final (and sometimes interim) report to account for the project development 
and the use of NLS funds. The accountability process focuses on the development of the 
proposal and the capacity of the organization to deliver on its project commitments. 

Organizations with funded projects welcome the flexibility of the NLS’ reporting 
requirements and most set up project steering committees or working groups to 
monitor their project. 

All twenty-one study organizations were satisfied with the flexibility of the NLS’ 
reporting requirements. They welcomed the freedom to implement their projects as they 
deemed fit and to monitor projects internally. Half of these organizations reported 
conducting some form of post project results assessment with project participants. Only 
one organization noted it did not have the time or resources to effectively monitor its 
projects on an ongoing basis. 

Over half of the case study organizations created some form of project steering 
committee, working group, or advisory board to monitor their project. These groups were 
generally appointed to provide assistance in keeping the projects on track and to provide 
advice, guidance, validation, and quality control of project outputs. 

Over half of the case study organizations submitted some form of interim report(s) to the 
NLS^ which were reviewed as part of the evaluation. The projects were required to 
provide a final report within three months of project completion; however, only one-third 
of organizations had submitted these reports at the time of this evaluation. Approximately 
half of the case study organizations had received timeline extensions, and had recently or 
had not yet completed their projects at the time of the evaluation. One project was 
terminated prematurely and was unable to submit a final report. 

One-quarter of the case study organizations reported that they had no contact with the 
NLS after project completion. All three of the case study organizations funded through 
the national funding stream indicated that the NLS was thorough and helpful. The NLS 
provided these organizations with ongoing feedback and advice to help guide the projects 
to successful completion. 



^ The information contained in the interim reports did not allow for a determination of impacts on individuals. 
Rather, as required, they only contained process information with respect to activities that had been implemented to 
achieve project objectives. A review of the documentation pertaining to post project assessments also did not allow 
for a determination of impacts on individuals. 
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Organization: College of New Caledonia 

Project: Community Planning for Intergenerational Learning 

Location: British Columbia 

In this community, no local agency had the mandate, staff, funds, or time to conduct a 
community wide, intergenerational assessment of literacy needs, challenges, and barriers to 
participation. Every social and educational agency in the community agreed that there are 
pressing literacy needs to be addressed, and that such needs could be better addressed if an 
approach was coordinated among all local service providers. 

An Advisory Committee was established by the College of New Caledonia to provide 
guidance and direction to the project. This Advisory Committee is still intact with more than a 
dozen community groups represented. The Committee continues to meet several times a year, 
and the coordinator meets informally with many of the members on an ongoing basis to 
discuss local literacy needs, believing that “to keep the community involved in literacy issues, 
you need to meet on an ongoing basis”. 



The project survey results as indicated in Table 6 below support the conclusion of the 
case study analysis that the majority of organizations with funded projects have set up 
steering committees or working groups and provided interim reports. 



Table 6 

Survey of Funded Organizations 




Required as Part of Project 








Project 


Accountabiiity Measures 


Yes 


No 


Incomplete 


Interim or progress reports to NLS 


72% 


19% 


1% 


Final or financial report 


85% 


6% 


8% 


Ongoing monitoring 


77% 


19% 


1% 


Working group / steering committee 


83% 


8% 


- 


Summative evaluation 


83% 


8% 


7% 


Provincial or Territorial requirements 


38% 


51% 


- 


Source: Survey of Funded Projects 



The need to put in place additional mechanisms for monitoring and accountability 
was identified as an area for improvement. 

Departments are responsible for ensuring that effective financial and program controls 
are implemented and that departmental capacity exists to effectively deliver and 
administer the grants, including monitoring (Treasury Board Transfer Payments Policy). 
More than half of the provincial and territorial key informants felt that there is 
insufficient program monitoring and accountability on the FTP funding stream to ensure 
the quality of products. For example, regarding funded research, it was noted that there 
are no quality assurances that the research meets established standards. 
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Key informants from funded projects confirmed that monitoring by the NLS usually 
occurs during the proposal stage when organizations are asked to describe their 
evaluation method for determining the success of the project. It was noted by less than 
half of those consulted that the NLS followed up according to an agreed-upon evaluation 
plan as described in the proposal. In about half the cases, it was noted that the NLS 
followed up by phone to see how the projects are doing. In 25% of the cases, the NLS 
conducted on-site visits to the project sponsors. 

The focus group of like organizations provided comments on evaluation and monitoring 
related to their experiences. They noted that graduated reporting requirements based on 
the size of the project can be helpful, and that certain sectors, such as the voluntary 
sector, often find the results-based model of accountability used by the federal 
government to be a foreign way of operating. They recommended that organizations such 
as the NLS should spend time educating and informing fund recipients and partners of the 
need to take this perspective of the results-based model of accountability and to move 
away from activity-based reporting. 



6.4 Dissemination of Information 

The dissemination of project results among stakeholders is an area where 
improvements have been made. There is a need to address this area further. 

The need to improve information dissemination and its potential for increasing the 
possibility of overlap and duplication were highlighted in the 1995 evaluation. Case 
study and key informant interviews in the current evaluation confirm that there has been 
an improvement in this area in the intervening period. The Secretariat acknowledges that 
while it has undertaken several activities to improve information dissemination, more 
should be done. 

Over all, one-third of the case study organizations felt the NLS could improve its efforts 
at coordinating and disseminating information and project results across the country. 
These organizations did not feel they had an adequate sense of literacy developments in 
other regions because information had not been effectively communicated, and therefore 
they had little insight into the NLS objectives and activities. Several of the organizations 
noted that they felt personally responsible for disseminating results and lessons learned to 
other literacy practitioners and interested parties and did not expect support from the NLS 
in this area. 
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Organization: Malaspina College University 

Project: How do Adults with Little Formal Education Learn? 

Location: British Columbia 

Research studies indicated that current adult basic education and literacy programs serve only a 
small number of people perceived to be ‘in need’ ’ . Literacy practitioners realize, that in spite of 
these statistics, adults do learn outside of the formal system. The question this project tried to 
answer was how? To answer this question, this project was designed to examine the when, 
where, what, and why these adults are learning. This information is expected to help literacy 
practitioners think of new and different ways in which to support learners. 

One of the NLS’s means of information sharing is through funding conferences. In the 
Malaspina College case, project representatives were invited to present the results of their 
project at several conferences, which they felt was helpful in sharing their lessons learned 
with the broader literacy community. 



Organization: Northwest Territories Literacy Council (NWTLC) 

Project: Family and Community Literacy Development Project 
Location: Northwest Territories 

The objective of the NWT Family and Community Literacy Development Project was to 
increase the NWTLC’ s relationships with its communities and to enhance the capacity of 
individuals and organizations to address literacy issues and needs within their communities, j 
The NWTLC delivered a three-day workshop in four communities where participants y 
identified the literacy needs of their community, created a plan for implementing a literacy j 
initiative, and then developed their ability to access various sources of funding. ( 

The NWTLC also developed and delivered 20 stand-alone workshops in seven NWT 
communities. The stand-alone workshops were used to create awareness surrounding the . 
benefits of family literacy and to bring people together to explore possible literacy activities ! 
and build skills. The NWTLC also facilitated a conference in Yellowknife that brought ^ 
together community workers from across the territory to discuss the possible links between 
family literacy and aboriginal language development. 

Provincial/territorial key informants interpreted information dissemination in two ways: 

• Passing on information to them; and 

• Informing stakeholders and others of the results of projects. 

As noted earlier (Section 5.2), the majority of these key informants indicated the need for 
better systems for tracking projects that have been funded, either through the NLS or the 
provinces and territories. At the proposal stage, for example, they emphasized that there 
was a need to identify whether a similar project has been funded elsewhere in Canada to 
ensure that projects selected for funding either build on the current literature/findings or 
take a new direction. Comments made by key informants from organizations with funded 
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projects suggest that the majority of the funded organizations consulted do not widely use 
the National Adult Literacy Database. 

The majority of the provincial and territorial key informants indicated that 
communication across the country has increased significantly over the last five years 
through the activities of the NLS, leading to the dissemination of innovative ideas 
throughout the literacy community. In particular, provincial and territorial representatives 
highly value the FPT meetings that are funded and supported by the NLS and credit this 
initiative with increasing the sharing of information, best practices and lessons learned 
across Canada. 



6.5 Overlap and Duplication 

There were two dimensions of enquiry regarding the risk of overlap and duplication. 
One referred to the potential for overlap and duplication of projects, while the other 
referred to the potential for overlap and duplication in the role played by the NLS itself. 

Program safeguards are needed to reduce the risk of project overlap and duplication. 

Program safeguards for reducing the risk of overlap and duplication, such as better 
systems for tracking projects, improved dissemination of information and improved 
information sharing with NLS partners were identified by key informants as ways to help 
ensure that a research project in one part of the country is not duplicated in another. Some 
project safeguards are in place to reduce the risk of overlap and duplication, such as the 
requirement for project sponsors to explain the rationale for funding their project and to 
review the National Adult Literacy Database (NALD) to ensure that the project is not 
duplicative; however, key informants from organizations with funded projects indicated that 
the majority of the funded organizations consulted do not widely use the NALD, suggesting 
that improvements can be made in this area (see Section 6.4). Key informants identified that 
the use of a peer review process during the application process would help ensure that NLS 
funded research projects and outputs meet the standards of the literacy community. Peer 
review processes were also discussed in Section 6.2 as a means of ensuring quality control 
of research projects. 

The case study analysis and key informants suggest that there is lack of knowledge 
regarding the different projects funded by the NLS and the outputs/outcomes of these 
projects. As a result, a majority of key informants believe that there is a risk of overlap or 
duplication in projects. Case studies indicate that the risk arises from a current lack of formal 
or strategic dissemination of information regarding the results of previous projects and a lack 
of information sharing on project proposals under consideration (see Section 6.4). 

Over half of the case study organizations stated that the NLS is not effectively 
communicating information about projects that have taken place, or those that are 
currently underway in various provinces among the funded projects. Consequently, it was 
difficult for them to comment on whether there is unnecessary duplication or overlap. 
One organization noted that even if similar projects were taking place in various provinces, 
each region and community is different and therefore this would not necessarily result in 
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overlap. One organization felt strongly, however, that the NLS was encouraging overlap 
because of its emphasis on new, ‘innovative’ ideas and developing new learning materials. 
This organization felt that efforts were being unnecessarily invested in developing new 
materials when there is a multitude of effective tools already available. 

On the other hand, the focus group of literacy experts believed that the NLS was doing a 
good job of letting people know about what had been funded across the country. 

In taking steps to address the risk of project overlap and duplication, the focus group of 
literacy experts warned that it would be important to bear in mind that, in a policy area 
such as literacy, one size does not fit all and that practitioners in different parts of the 
country doing similar projects does not mean there is duplication. For example, what 
some might see as duplication is more likely to be part of the ‘development’ process, with 
the replication of projects that have worked elsewhere providing value in its own right. 

With respect to the potential for NLS-sponsored applied research to be duplicative, it should 
be noted that the large scale national research undertaken with SSHRC and Statistics Canada 
is subject to their review mechanisms. Thus, there is little risk that it is duplicative. Research 
undertaken in the federal-provincial stream, including proposals that address meeting 
regional and community-based issues and needs, is subject to review by a federal-provincial 
committee. Research undertaken in the federal-provincial stream has a risk of duplicating 
work undertaken in other regions or communities. Thus there is a need for program 
safeguards to guard against the potential that this research duplicates work done in other 
regions/communities. Also, a review should be undertaken of needs assessment projects to 
determine if they are truly “applied research”. If not, they should be separately identified as a 
separate NLS funding category. 

The general view is that the NLS plays a unique role that is not being duplicated, 
although program safeguards are needed to demonstrate that projects and leveraged 
funds are incremental (i.e. would not have occurred without NLS support). 

Almost all key informants and representatives of the case study projects indicated that, as a 
catalyst and funding organization that works in partnership to achieve its objectives, the NLS 
plays a unique role that is not being duplicated. 

The project survey indicated that few (about 10%) of the organizations included in the 
survey considered there to be duplication between the NLS and other organizations in 
providing funds for literacy projects. Between one-quarter and one-third considered there 
to be overlap related to the other activities performed by the NLS (as shown in Table 7). 
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Table 7 

Overlap & Duplication 



Functions 


Aware of any Duplication? 


Yes 


No 


Overlap / Not 
really duplication 


Don’t know / 
no answer 


Funding (Q.8) 


8.7% 


89.9% 


0.7% 


0.7% 


Development of Learning Materials 
and Methods (Q.9a) 


25.6% 


60.3% 


3.2% 


10.8% 


Access and Outreach (Q.9b) 


22% 


66.1% 


2.5% 


9.4% 


Promotion and Public Awareness 
(Q9c) 


33.9% 


53.4% 


2.2% 


10.5% 


Coordination and Information 
Sharing (Q.9d) 


26.7% 


62.5% 


1 .8% 


9.0% 


Applied Research (Q.9e) 


23.1% 


62.5% 


0.7% 


13.7% 



The findings on dupiication by NLS activity as presented in Tabie 7 above are based on a survey of funded 
project sponsors. As such, they indicate that there is a perception among project sponsors of dupiication. 



Source: Survey of funded projects 



All provincial and territorial key informants considered there to be no overlap or 
duplication with provincial/territorial counterparts in terms of project funding. At the 
same time, however, the discussion on leveraging in Section 4.5 clearly indicates 
the need for the NLS to have program safeguards in place to demonstrate that NLS 
projects and leveraged funds are incremental (i.e. would not have occurred without the 
NLS). This is all the more important because of the limited evidence currently available 
and the ambiguity of such evidence. 
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7. Rationale and Relevance 



This section examined the rationale and relevance of the National Literacy Secretariat (NLS). 

The five activity/project areas used to guide NLS funding are still considered to be 
relevant. 

Three-quarters of the case study organizations suggested that there will always be a need for 
project funding at a national level. All of the case study organizations considered the 
five activity/project categories to be still relevant. They also considered the five categories to 
be broad enough to encompass most projects that seek to improve literacy in Canada. 

There may be an opportunity to develop a more strategic approach through the 
setting of focused priorities for the NLS. 

Some case study organizations suggested that some literacy areas might deserve a more 
focused effort. For example, one organization indicated that the NLS should make the 
direct support of program delivery a greater priority than research, as research has a more 
indirect impact on the literacy field. One organization indicated that the NLS is ideally 
situated to support projects related to nationwide leadership in literacy. Two case study 
organizations mentioned the need to promote literacy on a political level. One organization 
felt the NLS could have a greater impact by influencing policy in areas such as moving adult 
literacy to the forefront of Human Resources Development Canada (HRDC) priorities. As 
well, one organization felt the NLS should have a more visible presence promoting literacy 
issues and liaising between literacy stakeholders and provinces. 

In Quebec, the FPT agreement limits project funding to $25,000, which is viewed as 
restricting the scope of projects that can take place throughout the province. One organization 
suggested that the NLS distribute more funding to fewer projects in order to ensure the 
greatest impact, and to focus more on building on past experiences and best practices in order 
to achieve sustainable results. 

A majority of the provincial key informants felt that the NLS could assume a more 
strategic and national role in moving forward the literacy agenda and related priorities. They 
believe that the NLS should expand its leadership role of bringing together provincial and 
territorial counterparts to work together toward national goals and outcomes. In the years to 
come it was suggested that the NLS gives consideration to technological literacy, numeracy 
literacy, literacy among seniors, and Aboriginal literacy. 

When asked about the continued relevance of the NLS, at least one third of provincial 
and territorial respondents indicated that, from their perspective, literacy had a limited 
profile in the federal skills and learning agenda. As a result, they questioned whether the 
NLS should play a larger role within this initiative. 
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The majority of the key informants from organizations with funded projects indicated that 
the NLS, as a government sponsored program, should lead a national campaign on 
literacy. It was also recommended that the NLS could be a more effective information 
conduit for individuals and organizations working in the literacy field across Canada. 

The development of a more strategic approach to the NLS should include the 
consideration of ways to increase the sustainability of NLS funded projects and results. 

Three-quarters of the case study organizations suggested that the NLS should consider 
providing ongoing core program funding. One organization from the national funding 
stream noted that it is much easier to raise funds from corporate sponsors for high profile 
projects than it is to obtain core funding. 

All of the case study organizations expressed concern that restrictions in NLS funding 
related to one-year funding cycles and a directive to fund ideas that are innovative and at 
a development stage (i.e., not ongoing) was impeding the sustainability of literacy 
projects. While the focus on innovation is viewed in a positive light, the need to develop 
new ideas and project proposals annually is seen as cumbersome and redundant. Over 
three-quarters of the case study organizations feel that NLS funds would be best invested 
in promoting, expanding, and building upon past successes. 



Organization: Comite Alpha Papineau 

Project: L’Alphabetisation par les pairs (Literacy by Peers) 

Location: Quebec 

Consultations indicated that community members required assistance in filing tbeir income 
taxes, which is a service CAP offers. However, these consultations revealed that the income 
tax specialist they hired in the past was often intimidating to clients. Therefore, CAP 
developed L’Alphabetisation par les pairs project, which was designed to hire an income tax 
specialist who could train a select group of clients in filing taxes, who would then in turn 
mentor their peers in developing the skills and knowledge to file their income taxes. 

This literacy mentorship project received coverage in the local papers and on television, 
which CAP believes is linked to the increased number of volunteers it has received for other 
literacy related projects. While the L’Alphabetisation par les pairs project was viewed as very 
successful, CAP did not have the funding to implement the program on an ongoing basis. 
In fact, with the conclusion of NLS funding, the project was terminated, as there was no 
further funding to hire a trainer to continue it the following year. The termination of the 
project was viewed as diminishing the long-term impact of the project. 
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Organization: Street Haven Learning Centre 

Project: Street Reach: Building the Capacity to do Outreach 

Location: Ontario 

This project was aimed at influencing literacy programs and social service agencies who work 
with people who are homeless, potential literacy learners who are homeless, and the literacy 
field hy increasing the level of awareness of literacy issues in the social service field and 
developing tools and practices to assist outreach efforts. 

An important aspect of sustainability is capacity building, either at the program level or among 
learners. One of the positive outcomes of this project has been the enhancement of effective 
partnerships, which have allowed for the sharing of resources and knowledge related to effective 
outreach practices. The referral protocols and marketing materials are expected to help sustain 
Street Haven’s, and similar organizations’, capacity to do outreach in the long-term. Throughout 
the project, the knowledge that has been gained and tools that were developed have been shared 
with the larger literacy community through AlphaPlus and NALD, as well as through more 
informal information requests generating from word of mouth. 



The issue of sustainability of projects was also identified by the provincial and territorial 
key informants. All believe that there is a real need for NLS to consider funding in 
support of ongoing projects. The majority noted that provinces and other stakeholders do 
not have the capacity to provide all the necessary resources to sustain many projects after 
the start-up phase. They also noted that valuable start-up projects often do not reach their 
potential due to lack of ongoing NLS funding. 

Representatives from approximately one third of provinces and territories suggested that 
one approach would be for the NLS to consider giving funds directly to the provinces and 
territories to distribute. However, one province noted that if the NLS withdrew from 
funding projects and provided money directly to the province, literacy spending would 
likely be discontinued within the province. 




The focus group of literacy experts noted that it is time to revisit the NLS mandate and 
objectives. It was argued that although they believed the NLS was doing a good job, the 
Secretariat must take into account that society has changed. With globalization, the creation 
of the knowledge economy and the changing patterns of immigration, there is now a need to 
focus on lifelong learning. There is also a need for the NLS to become more proactive in 
influencing policy in order to develop a national literacy agenda. The focus group of l ik e 
organizations advised the NLS to identify and draw resources from across government and 
communities in order to increase its effectiveness. 
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8. Summary of Findings/ 
Areas for Improvement 



8.1 Evaluation Findings 

Impacts and effects : There is evidence that National Literacy Secretariat (NLS) 
funding has improved literacy opportunities for a range of groups. 

The case study analysis and key informants indicated that the NLS has had an important 
impact in areas such as family and workplace literacy. The project survey indicated that NLS 
funded projects improved literacy opportunities for a range of groups including low income 
people, people with disabilities, first generation Canadians, and Aboriginal people. 

A catalyst ; The general view is that the NLS acts as a catalyst to increase awareness 
of literacy issues, bring people together and advance literacy. 

Evidence from case studies, key informant interviews, and the focus group of literacy 
experts indicates that the NLS has raised the profile of literacy and helped to increase 
collaboration/partnerships within the literacy community. The available evidence also 
suggests that the NLS has advanced the literacy movement by promoting research and 
improving the ability to deliver successful literacy programs within the community. 

Also, there is a high level of satisfaction with 70% of the organizations surveyed 
indicating that they were either satisfied or very satisfied with NLS services. 

Partnership model ; Regarding the FPT funding stream, the partnership model has 
been functioning well in the Canadian inter-jurisdictional environment. The NLS 
should consider, however, improving communication with partners in relation to the 
selection and announcement of projects funded under the national funding stream. 

All provincial and territorial stakeholders, as well as funded organizations familiar with 
the NLS and its activities, were satisfied with the functioning of the partnership model as 
it applied to the Lederal-provincial-territorial (LPT) funding stream. Lor example, they 
cited that it facilitated flexibility and openness. While the NLS partnership model is 
viewed as working well under the LPT program, provincial and territorial partners would 
like to see better communications and working relations with the NLS in the case of the 
program’s national funding stream. 

Project funding ; NLS funding is broadly dispersed across many organizations. 

Over 1997-2002 period, 2,043 organizations received NLS funding for 2,853 projects. 
Most organizations (80%) had only one project during the period. In terms of funding 
support, 50% of all projects funded by the NLS received less than $25,000. 
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8.2 Areas for Improvement 

Areas for improvement identified in the 1995 evaluation included the need for clearly 
defined priorities concerning issues to be pursued and projects funded, a more active role 
in ensuring the quality of research and learning materials produced and improved 
dissemination of results in order to safeguard against project duplication. Although 
program enhancements have occurred since 1995, (e.g. improved communication) there 
is still room for improvement (e.g. monitoring of results). 

Leveraging ; There is some evidence that NLS support results in organizations being 
able to leverage further resources to support literacy efforts. However, there is a 
need for the NLS to be able to demonstrate that it has program safeguards in place 
to ensure that NLS projects and leveraged funds are incremental. 

The majority of organizations that receive funding from the NLS also receive funding 
and/or in-kind resources from project partners and over time the share of total project 
funds accounted for by partners’ contributions is declining. The program needs 
safeguards in place to protect against projects being funded that would have proceeded 
regardless of NLS funding. A potential safeguard could require sponsors to attest on the 
project application that the project would not proceed without NLS support. 

Dissemination of information ; While the NLS is credited for supporting the National 
Adult Literacy Database (NALD) and International Adult Learning & Skills Survey 
(lALSS) as well as regular intergovernmental consultation, more attention should be 
given to ensuring dissemination of project results among stakeholders. 

The majority of the provincial and territorial key informants indicated that 
communication across the country has increased significantly over the last five years 
through the activities of the NLS, leading to the dissemination of innovative ideas 
throughout the literacy community; however; evidence from the case studies and key 
informants indicate that there is a need to further improve the dissemination of 
information and results to stakeholders and partners. 

The financial support for the NALD and the lALSS provided by the NLS reflects the 
commitment of the NLS to improve communication and sharing of data. At the same 
time, however, the evidence indicates that there is a need to further improve the 
dissemination of information and results to stakeholders and partners. 

Overlap and duplication ; The general view is that as a catalyst and an organization that 
works in partnership to achieve its objectives, the NLS plays a unique role that is not 
being duplicated. There is a risk, however, of duplication in specific project support. 

There is a risk of project overlap and duplication, particularly in the application of the FPT 
funding stream. Evidence from case studies, key informants and survey respondents indicate 
a risk of duplication in specific project support. This risk arises from the current lack of 
formal or strategic dissemination of information regarding the results of previous projects, 
and a current lack of information sharing on project proposals that are under consideration. 
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Increased sharing of information with NLS partners was identified as a way to help ensure 
that a research project in one part of the country is not duplicated in another. 

Program implementation ; The NLS could simplify and streamline procedures to 
make it easier for smaller organizations less familiar with the grant process to 
obtain funds. 

Groups familiar with the NLS and its application process feel that process and guidelines 
are clear and work well. Organizations that are less fa mi liar with the NLS and its work 
find that application and notification procedures are cumbersome, that forms are 
complex, and that the need to develop new annual applications and approaches makes it 
difficult for smaller organizations to be successful in the competition for project funding. 

Quality : The NLS should take steps to ensure the reliahility and credibility of 
research projects. 

While major research projects conducted through SSHRC and Statistics Canada ensure 
high quality and reduce the risk of being duplicative through peer review mechanisms, 
FPT research runs the risk of duplicating work done in other regions/communities. 
In addition, more than one-half of the provincial and territorial key informants felt that there 
is insufficient program monitoring and accountability on the FTP funding stream to ensure 
the quality of products. Furthermore, needs assessment projects funded under the FPT 
stream may lack the rigour necessary to be considered ‘applied research’. The application of 
a formal peer review process for all research conducted would help increase quality and help 
reduce the risk of duplication. 

Monitoring performance : The NLS should put in place efficient data gathering and 
monitoring systems and consistent procedures for evaluating projects. 

The monitoring of the results of projects by the NLS was identified as an area for 
improvement, even given the limitations imposed by the grant process. Also, there is a 
need to improve the data gathering and monitoring system (GMAX) used for the 
administration of grants. Coding limitations restrict its usefulness in providing an in 
depth perspective of NLS activities and outputs. Data collection and storage should be 
examined to ensure that data is available to conduct a future evaluation. 

While the NLS is credited with being a catalyst in promoting literacy across Canada, it is 
unclear to what extent it has been successful in meeting its stated objectives, due in part 
to issues of attribution and measurement. 

Relevance : While the NLS plays a unique role in promoting literacy across Canada, 
there may be an opportunity to develop a more strategic approach through the 
setting of focused priorities. 

There is a concern as to what the organization’s role in the future should be relative to its 
current objectives and the needs of the literacy community. One approach is for the NLS 
to ‘stay the course’. This approach reflects the view that more work needs to be done in 
raising the profile of literacy, and that the NLS as a funding organization continues to 
play a unique role in supporting the attainment of its objectives through other 
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organizations. Another approach is for the NLS to consider developing a national 
approach to literacy that would involve the setting of Canada-wide priorities that would 
reflect the specific needs of today (e.g. focusing on technology, focusing on the needs of 
specific groups such as Aboriginals and families). 

This suggests that there is an opportunity for the NLS to re-examine its role in order to 
determine how it could best meet its objectives in the changing environment. This could 
include a consultation process with stakeholders across the country to assess how the 
literacy challenge has evolved over the last fourteen years. 

Sustainability and NLS Role ; As part of developing a more strategic approach, the 
NLS may wish to look at ways to increase the sustainability of results and progress. 

The NLS is the primary literacy organization in Canada which funds and supports 
first-time projects. More than half of the NLS funded projects continued after program 
funding expired, however, almost one-third of NLS funded projects were not sustainable. 
There are some pressures, however, for the NLS to consider providing longer-term 
funding in order to promote, expand and build upon past successes and to improve on 
project sustainability. While this could raise inter-jurisdictional issues, such an approach 
could be considered as part of a re-examination of the program’s role. 
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National Literacy Secretariat Logic Model 

Objectives 



Appendix I 
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Appendix II 

Case Study Summary: National Literacy Secretariat 



This appendix provides a brief summary of the activities and results of selected projects 
(those that are part of the case study analysis) funded by the NLS. The case studies 
reviewed 21 projects comprising: developing learning materials (6); improving coordination 
and information sharing (4); improving access to literacy programs and outreach (4); 
increasing public awareness of literacy issues (4); and research (3). For each case study the 
summary includes information from the Assessment Reports provided by funded 
organizations to the NLS. Specifically included are the participating organizations’ 
comments regarding the funding stream, project overview, intended client groups, project 
objectives, degree to which project objectives were met, barriers / factors affecting objective 
achievement, expected impacts, and project status. 

Information (on impacts and objectives achievement) is based on the participating 
organizations’ self-assessments. Most projects ended or were scheduled to end by 2002. 

The NLS funds projects that involve only intermediate outputs that are in support of direct 
literacy training. It does not itself provide funding for direct literacy instruction to individual 
user-clients. There are many factors that come into play at the level of individual instruction 
for the user-clienteles involved. The outputs accompanying NLS interventions are but a 
subset of these. Therefore, attribution on the basis of the program’s interventions with 
respect to individual project/user-client impacts is very difficult, and in some cases, not 
feasible at all. In addition, there is the point that the evaluation found that, in many instances 
NLS interventions/support is funded through grant mechanisms rather than contributions. 
Data collection in these instances, particularly with respect to follow-up/outcomes analysis of 
program results, is virtually non-existent. As a result of all of the above, the evaluation was 
unable to utilize standard methodologies based on follow-up and attributions. Consequently, 
assessments of the program’s impact in these circumstances is required to rely on indirect 
evidence which is only general and very approximate in nature. 
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